2022-11-23T01:30:12.1712992Z Requested labels: linux.rocm.gpu 2022-11-23T01:30:12.1713066Z Job defined at: pytorch/pytorch/.github/workflows/_rocm-test.yml@refs/heads/master 2022-11-23T01:30:12.1713089Z Waiting for a runner to pick up this job... 2022-11-23T01:30:12.3626990Z Job is about to start running on the runner: worker-rocm-amd-94 (repository) 2022-11-23T01:30:15.4891212Z Current runner version: '2.299.1' 2022-11-23T01:30:15.4896876Z Runner name: 'worker-rocm-amd-94' 2022-11-23T01:30:15.4897404Z Runner group name: 'Default' 2022-11-23T01:30:15.4898179Z Machine name: 'jenkins-worker-rocm-amd-94' 2022-11-23T01:30:15.4900491Z ##[group]GITHUB_TOKEN Permissions 2022-11-23T01:30:15.4901148Z Actions: write 2022-11-23T01:30:15.4901468Z Checks: write 2022-11-23T01:30:15.4901793Z Contents: write 2022-11-23T01:30:15.4902096Z Deployments: write 2022-11-23T01:30:15.4902434Z Discussions: write 2022-11-23T01:30:15.4902842Z Issues: write 2022-11-23T01:30:15.4903152Z Metadata: read 2022-11-23T01:30:15.4903483Z Packages: write 2022-11-23T01:30:15.4903776Z Pages: write 2022-11-23T01:30:15.4904117Z PullRequests: write 2022-11-23T01:30:15.4904486Z RepositoryProjects: write 2022-11-23T01:30:15.4904860Z SecurityEvents: write 2022-11-23T01:30:15.4905193Z Statuses: write 2022-11-23T01:30:15.4905486Z ##[endgroup] 2022-11-23T01:30:15.4908891Z Secret source: Actions 2022-11-23T01:30:15.4909491Z Prepare workflow directory 2022-11-23T01:30:15.8815037Z Prepare all required actions 2022-11-23T01:30:15.9029431Z Getting action download info 2022-11-23T01:30:16.2010238Z Download action repository 'pytorch/pytorch@master' (SHA:1cfd3858ac54fe3883534309081631a0a892ba3f) 2022-11-23T01:30:22.9968547Z Download action repository 'pytorch/test-infra@main' (SHA:c57ff4d9a93667a5571a80a0e92c3e2674aeedfd) 2022-11-23T01:30:24.1986976Z Getting action download info 2022-11-23T01:30:24.4478795Z Download action repository 'malfet/checkout@silent-checkout' (SHA:c7b8fef48edfe1bca0044a44b1f7f7c4318a3076) 2022-11-23T01:30:25.4516761Z Uses: pytorch/pytorch/.github/workflows/_rocm-test.yml 2022-11-23T01:30:25.4518589Z ##[group] Inputs 2022-11-23T01:30:25.4518934Z build-environment: linux-focal-rocm5.2-py3.8 2022-11-23T01:30:25.4519448Z test-matrix: { include: [ { config: "distributed", shard: 1, num_shards: 2, runner: "linux.rocm.gpu" }, { config: "distributed", shard: 2, num_shards: 2, runner: "linux.rocm.gpu" }, ]} 2022-11-23T01:30:25.4520098Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm5.2-py3.8:072aae4a77ed7d3a69ad5683420509c41301b940 2022-11-23T01:30:25.4520516Z sync-tag: 2022-11-23T01:30:25.4520748Z ##[endgroup] 2022-11-23T01:30:25.4521435Z Complete job name: linux-focal-rocm5.2-py3.8-distributed / test (distributed, 1, 2, linux.rocm.gpu, mem_leak_check) 2022-11-23T01:30:25.5760665Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@master 2022-11-23T01:30:25.5761048Z with: 2022-11-23T01:30:25.5761289Z no-sudo: true 2022-11-23T01:30:25.5761558Z submodules: recursive 2022-11-23T01:30:25.5761800Z fetch-depth: 0 2022-11-23T01:30:25.5762050Z env: 2022-11-23T01:30:25.5762299Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:30:25.5762557Z ##[endgroup] 2022-11-23T01:30:25.6018833Z ##[group]Run retry () { 2022-11-23T01:30:25.6019162Z retry () { 2022-11-23T01:30:25.6019468Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2022-11-23T01:30:25.6019754Z } 2022-11-23T01:30:25.6020021Z echo "${GITHUB_WORKSPACE}" 2022-11-23T01:30:25.6020327Z if [ -z "${NO_SUDO}" ]; then 2022-11-23T01:30:25.6020649Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2022-11-23T01:30:25.6020939Z else 2022-11-23T01:30:25.6021252Z  retry rm -rf "${GITHUB_WORKSPACE}" 2022-11-23T01:30:25.6021543Z fi 2022-11-23T01:30:25.6021811Z mkdir "${GITHUB_WORKSPACE}" 2022-11-23T01:30:25.6049601Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:30:25.6049934Z env: 2022-11-23T01:30:25.6050200Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:30:25.6050452Z NO_SUDO: true 2022-11-23T01:30:25.6050705Z ##[endgroup] 2022-11-23T01:30:25.6345331Z /home/pytorchci/actions-runner/_work/pytorch/pytorch 2022-11-23T01:30:27.1891854Z ##[group]Run malfet/checkout@silent-checkout 2022-11-23T01:30:27.1892253Z with: 2022-11-23T01:30:27.1892560Z ref: 1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T01:30:27.1893085Z fetch-depth: 0 2022-11-23T01:30:27.1893369Z submodules: recursive 2022-11-23T01:30:27.1893664Z quiet-checkout: true 2022-11-23T01:30:27.1893995Z repository: pytorch/pytorch 2022-11-23T01:30:27.1894477Z token: *** 2022-11-23T01:30:27.1894772Z ssh-strict: true 2022-11-23T01:30:27.1895090Z persist-credentials: true 2022-11-23T01:30:27.1895408Z clean: true 2022-11-23T01:30:27.1895664Z lfs: false 2022-11-23T01:30:27.1895954Z set-safe-directory: true 2022-11-23T01:30:27.1896240Z env: 2022-11-23T01:30:27.1896512Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:30:27.1896792Z ##[endgroup] 2022-11-23T01:30:27.3467766Z Syncing repository: pytorch/pytorch 2022-11-23T01:30:27.3469342Z ##[group]Getting Git version info 2022-11-23T01:30:27.3469981Z Working directory is '/home/pytorchci/actions-runner/_work/pytorch/pytorch' 2022-11-23T01:30:27.3470604Z [command]/usr/bin/git version 2022-11-23T01:30:27.3470864Z git version 2.35.1 2022-11-23T01:30:27.3471584Z ##[endgroup] 2022-11-23T01:30:27.3484521Z Temporarily overriding HOME='/home/pytorchci/actions-runner/_work/_temp/58864c16-8173-4ea3-92d3-803d8cf926b2' before making global git config changes 2022-11-23T01:30:27.3485081Z Adding repository directory to the temporary git global config as a safe directory 2022-11-23T01:30:27.3485726Z [command]/usr/bin/git config --global --add safe.directory /home/pytorchci/actions-runner/_work/pytorch/pytorch 2022-11-23T01:30:27.3501613Z Deleting the contents of '/home/pytorchci/actions-runner/_work/pytorch/pytorch' 2022-11-23T01:30:27.3510041Z ##[group]Initializing the repository 2022-11-23T01:30:27.3515657Z [command]/usr/bin/git init /home/pytorchci/actions-runner/_work/pytorch/pytorch 2022-11-23T01:30:27.3569097Z hint: Using 'master' as the name for the initial branch. This default branch name 2022-11-23T01:30:27.3570207Z hint: is subject to change. To configure the initial branch name to use in all 2022-11-23T01:30:27.3571116Z hint: of your new repositories, which will suppress this warning, call: 2022-11-23T01:30:27.3571731Z hint: 2022-11-23T01:30:27.3572523Z hint: git config --global init.defaultBranch 2022-11-23T01:30:27.3573077Z hint: 2022-11-23T01:30:27.3573822Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2022-11-23T01:30:27.3575040Z hint: 'development'. The just-created branch can be renamed via this command: 2022-11-23T01:30:27.3576245Z hint: 2022-11-23T01:30:27.3576897Z hint: git branch -m 2022-11-23T01:30:27.3577893Z Initialized empty Git repository in /home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/ 2022-11-23T01:30:27.3594059Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2022-11-23T01:30:27.3649216Z ##[endgroup] 2022-11-23T01:30:27.3650375Z ##[group]Disabling automatic garbage collection 2022-11-23T01:30:27.3659619Z [command]/usr/bin/git config --local gc.auto 0 2022-11-23T01:30:27.3716411Z ##[endgroup] 2022-11-23T01:30:27.3717480Z ##[group]Setting up auth 2022-11-23T01:30:27.3735113Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2022-11-23T01:30:27.3790189Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || : 2022-11-23T01:30:27.4118250Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2022-11-23T01:30:27.4173404Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || : 2022-11-23T01:30:27.4561417Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2022-11-23T01:30:27.4643715Z ##[endgroup] 2022-11-23T01:30:27.4645018Z ##[group]Fetching the repository 2022-11-23T01:30:27.4662824Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --quiet --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2022-11-23T01:31:27.9408046Z [command]/usr/bin/git rev-parse --verify --quiet 1cfd3858ac54fe3883534309081631a0a892ba3f^{object} 2022-11-23T01:31:27.9435669Z 1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T01:31:27.9444414Z ##[endgroup] 2022-11-23T01:31:27.9445752Z ##[group]Determining the checkout info 2022-11-23T01:31:27.9447475Z ##[endgroup] 2022-11-23T01:31:27.9448755Z ##[group]Checking out the ref 2022-11-23T01:31:27.9456000Z [command]/usr/bin/git checkout --quiet --force 1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T01:31:29.2491877Z ##[endgroup] 2022-11-23T01:31:29.2493184Z ##[group]Setting up auth for fetching submodules 2022-11-23T01:31:29.2501170Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2022-11-23T01:31:29.2585795Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2022-11-23T01:31:29.2652383Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2022-11-23T01:31:29.2705445Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2022-11-23T01:31:29.2749431Z ##[endgroup] 2022-11-23T01:31:29.2750663Z ##[group]Fetching submodules 2022-11-23T01:31:29.2756278Z [command]/usr/bin/git submodule sync --recursive 2022-11-23T01:31:29.3147290Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2022-11-23T01:31:29.3459833Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2022-11-23T01:31:29.3462932Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2022-11-23T01:31:29.3465222Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2022-11-23T01:31:29.3467439Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2022-11-23T01:31:29.3469542Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK' 2022-11-23T01:31:29.3472326Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2022-11-23T01:31:29.3475596Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2022-11-23T01:31:29.3478043Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2022-11-23T01:31:29.3480700Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2022-11-23T01:31:29.3484095Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub' 2022-11-23T01:31:29.3487906Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2022-11-23T01:31:29.3492070Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2022-11-23T01:31:29.3495035Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2022-11-23T01:31:29.3499012Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2022-11-23T01:31:29.3503254Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2022-11-23T01:31:29.3506127Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2022-11-23T01:31:29.3510062Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2022-11-23T01:31:29.3514261Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2022-11-23T01:31:29.3518353Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2022-11-23T01:31:29.3522566Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2022-11-23T01:31:29.3526815Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2022-11-23T01:31:29.3531663Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake' 2022-11-23T01:31:29.3537077Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2022-11-23T01:31:29.3541419Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2022-11-23T01:31:29.3545916Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2022-11-23T01:31:29.3550543Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse' 2022-11-23T01:31:29.3555292Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2022-11-23T01:31:29.3560060Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2022-11-23T01:31:29.3565149Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt' 2022-11-23T01:31:29.3570494Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2022-11-23T01:31:29.3575507Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2022-11-23T01:31:29.3580685Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2022-11-23T01:31:29.3586032Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2022-11-23T01:31:29.3591149Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2022-11-23T01:31:29.3596880Z Submodule 'third_party/python-enum' (https://github.com/PeachPy/enum34.git) registered for path 'third_party/python-enum' 2022-11-23T01:31:29.3602235Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2022-11-23T01:31:29.3608032Z Submodule 'third_party/python-six' (https://github.com/benjaminp/six.git) registered for path 'third_party/python-six' 2022-11-23T01:31:29.3613491Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2022-11-23T01:31:29.3619574Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb' 2022-11-23T01:31:29.3625472Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2022-11-23T01:31:29.3631438Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd' 2022-11-23T01:31:29.3726982Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2022-11-23T01:31:30.4695885Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2022-11-23T01:31:31.3894870Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2022-11-23T01:31:32.2155981Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2022-11-23T01:31:33.3420943Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'... 2022-11-23T01:31:34.4145776Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2022-11-23T01:31:37.4725892Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2022-11-23T01:31:42.7470175Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2022-11-23T01:31:44.2434665Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2022-11-23T01:31:45.6269964Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cub'... 2022-11-23T01:31:47.8876353Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2022-11-23T01:31:50.2170792Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2022-11-23T01:31:52.5101396Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2022-11-23T01:31:57.4191871Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2022-11-23T01:31:59.6589955Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2022-11-23T01:32:04.6246683Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2022-11-23T01:32:07.0765282Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2022-11-23T01:32:07.9023333Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2022-11-23T01:32:09.5030156Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2022-11-23T01:32:10.7655028Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2022-11-23T01:32:12.5684007Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2022-11-23T01:32:13.8729556Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'... 2022-11-23T01:32:14.7490303Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2022-11-23T01:32:15.7393634Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2022-11-23T01:32:18.3658660Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2022-11-23T01:32:19.7837357Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'... 2022-11-23T01:32:20.9537306Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2022-11-23T01:32:27.8750546Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2022-11-23T01:32:30.4703175Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'... 2022-11-23T01:32:31.8120067Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2022-11-23T01:32:32.8201380Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2022-11-23T01:32:38.5777663Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2022-11-23T01:32:39.3018086Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2022-11-23T01:32:40.2919024Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2022-11-23T01:32:42.1910107Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/python-enum'... 2022-11-23T01:32:43.0141174Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2022-11-23T01:32:44.2061144Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/python-six'... 2022-11-23T01:32:45.3872221Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2022-11-23T01:32:47.4186511Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tbb'... 2022-11-23T01:32:50.5047881Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2022-11-23T01:32:51.8956434Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/zstd'... 2022-11-23T01:32:54.7532770Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2022-11-23T01:32:54.7904002Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2022-11-23T01:32:54.8259055Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2022-11-23T01:32:54.8809461Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2022-11-23T01:32:54.9351239Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c' 2022-11-23T01:32:54.9992379Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2022-11-23T01:32:55.6083584Z Submodule path 'third_party/XNNPACK': checked out 'ae108ef49aa5623b896fc93d4298c49d1750d9ba' 2022-11-23T01:32:55.6666781Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2022-11-23T01:32:55.7983351Z Submodule path 'third_party/cpuinfo': checked out '8ec7bd91ad0470e61cf38f618cc1f270dede599c' 2022-11-23T01:32:55.8667888Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4' 2022-11-23T01:32:56.1737668Z Submodule path 'third_party/cudnn_frontend': checked out '171a7a986f7fbd9ed71bd0cf3c7ad4f55843d6b3' 2022-11-23T01:32:56.5902235Z Submodule path 'third_party/cutlass': checked out 'b72cbf957df8cf84a6d0ff91c190ad51a9c1d24a' 2022-11-23T01:32:56.8610953Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2022-11-23T01:32:56.9450209Z Submodule path 'third_party/fbgemm': checked out '4d1738b3142a6cb0c032cd639e239566010b054a' 2022-11-23T01:32:56.9526180Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2022-11-23T01:32:56.9530358Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T01:32:56.9538243Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2022-11-23T01:32:56.9545925Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T01:32:56.9627927Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2022-11-23T01:32:58.7285033Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2022-11-23T01:33:00.1213363Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2022-11-23T01:33:02.3772544Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2022-11-23T01:33:03.4266664Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2022-11-23T01:33:03.5573976Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2022-11-23T01:33:03.6659881Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2022-11-23T01:33:03.7038990Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '1840658c184f3eeba787dae0f06c45756c1daaf5' 2022-11-23T01:33:03.8315216Z Submodule path 'third_party/flatbuffers': checked out 'd0cede9c90c5257537c293517a21376408b549fa' 2022-11-23T01:33:03.9007384Z Submodule path 'third_party/fmt': checked out '7bdf0628b1276379886c7f6dda2cef2b3b374f0b' 2022-11-23T01:33:03.9397993Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2022-11-23T01:33:04.0069716Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2022-11-23T01:33:04.0638376Z Submodule path 'third_party/gloo': checked out '4a5e339b764261d20fc409071dc7a8b8989aa195' 2022-11-23T01:33:04.1418635Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2022-11-23T01:33:04.1870012Z Submodule path 'third_party/ideep': checked out '5ddc65efe0428bbce2942b3ce5e3ce15239abe2f' 2022-11-23T01:33:04.1943308Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2022-11-23T01:33:04.2008144Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2022-11-23T01:33:12.7541429Z Submodule path 'third_party/ideep/mkl-dnn': checked out 'd19d0f795c60695bd32f894c6f01771b2dfbe24d' 2022-11-23T01:33:12.7624809Z Submodule 'third_party/oneDNN' (https://github.com/oneapi-src/oneDNN.git) registered for path 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T01:33:12.7716275Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN'... 2022-11-23T01:33:21.1541111Z Submodule path 'third_party/ideep/mkl-dnn/third_party/oneDNN': checked out '650085b2f3643aad05c629425983491d63b5c289' 2022-11-23T01:33:21.1923091Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724' 2022-11-23T01:33:21.2373353Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2022-11-23T01:33:21.3608208Z Submodule path 'third_party/kineto': checked out '6c1629809068efd78a8d56b4aa479c7ec49ae562' 2022-11-23T01:33:21.3682894Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T01:33:21.3688825Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T01:33:21.3762449Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2022-11-23T01:33:23.2865046Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2022-11-23T01:33:25.1772701Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '2591ab91c3898c9f6544fff04660276537d32ffd' 2022-11-23T01:33:25.2774133Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2022-11-23T01:33:25.3896827Z Submodule path 'third_party/nccl/nccl': checked out 'f89fd4777d2ef9229c039ff750ae21da01626f52' 2022-11-23T01:33:25.4395914Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a' 2022-11-23T01:33:25.5738180Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2022-11-23T01:33:25.8463074Z Submodule path 'third_party/onnx': checked out 'f7ee1ac60d06abe8e26c9b6bbe1e3db5286b614b' 2022-11-23T01:33:25.8572554Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2022-11-23T01:33:25.8575709Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2022-11-23T01:33:25.8688182Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2022-11-23T01:33:27.1646348Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2022-11-23T01:33:30.6592230Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2022-11-23T01:33:30.7278455Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'ffa346860b306c9bbfb341aed9c14c067751feb8' 2022-11-23T01:33:30.7770039Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f' 2022-11-23T01:33:30.7838647Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T01:33:30.7904650Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'... 2022-11-23T01:33:33.3431407Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8' 2022-11-23T01:33:33.3524804Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T01:33:33.3530540Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T01:33:33.3627194Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'... 2022-11-23T01:33:34.5660599Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'... 2022-11-23T01:33:36.2318433Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508' 2022-11-23T01:33:36.3443825Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c' 2022-11-23T01:33:36.3515982Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T01:33:36.3589690Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'... 2022-11-23T01:33:37.3195570Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2022-11-23T01:33:37.3588081Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a' 2022-11-23T01:33:37.6365127Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2022-11-23T01:33:37.6442095Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2022-11-23T01:33:37.6444316Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2022-11-23T01:33:37.6524199Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2022-11-23T01:33:39.1317906Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2022-11-23T01:33:41.0266588Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2022-11-23T01:33:41.1454557Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2022-11-23T01:33:41.1822081Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2022-11-23T01:33:41.2224659Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413' 2022-11-23T01:33:41.2908197Z Submodule path 'third_party/pybind11': checked out '80dc998efced8ceb2be59756668a7e90e8bef917' 2022-11-23T01:33:41.3294487Z Submodule path 'third_party/python-enum': checked out '4cfedc426c4e2fc52e3f5c2b4297e15ed8d6b8c7' 2022-11-23T01:33:41.3849872Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2022-11-23T01:33:41.4214220Z Submodule path 'third_party/python-six': checked out '15e31431af97e5e64b80af0a3f598d382bcdd49a' 2022-11-23T01:33:41.4961090Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff' 2022-11-23T01:33:41.6353799Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9' 2022-11-23T01:33:41.6983901Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2022-11-23T01:33:41.7056730Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2022-11-23T01:33:41.7061885Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2022-11-23T01:33:41.7068209Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2022-11-23T01:33:41.7074849Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T01:33:41.7149738Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2022-11-23T01:33:43.4236146Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2022-11-23T01:33:44.6830483Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2022-11-23T01:33:46.8084015Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2022-11-23T01:33:49.5040535Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2022-11-23T01:33:49.5427778Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2022-11-23T01:33:49.6483984Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2022-11-23T01:33:49.7129217Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2022-11-23T01:33:49.7203134Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T01:33:49.7267352Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2022-11-23T01:33:50.6663515Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2022-11-23T01:33:50.8514709Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8' 2022-11-23T01:33:50.8651290Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2022-11-23T01:33:50.9070270Z Entering 'android/libs/fbjni' 2022-11-23T01:33:50.9132199Z Entering 'third_party/FP16' 2022-11-23T01:33:50.9191402Z Entering 'third_party/FXdiv' 2022-11-23T01:33:50.9256605Z Entering 'third_party/NNPACK' 2022-11-23T01:33:50.9321380Z Entering 'third_party/QNNPACK' 2022-11-23T01:33:50.9391027Z Entering 'third_party/VulkanMemoryAllocator' 2022-11-23T01:33:50.9455336Z Entering 'third_party/XNNPACK' 2022-11-23T01:33:50.9544901Z Entering 'third_party/benchmark' 2022-11-23T01:33:50.9611607Z Entering 'third_party/cpuinfo' 2022-11-23T01:33:50.9675642Z Entering 'third_party/cub' 2022-11-23T01:33:50.9744071Z Entering 'third_party/cudnn_frontend' 2022-11-23T01:33:50.9821779Z Entering 'third_party/cutlass' 2022-11-23T01:33:50.9907404Z Entering 'third_party/eigen' 2022-11-23T01:33:50.9985585Z Entering 'third_party/fbgemm' 2022-11-23T01:33:51.0055111Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-11-23T01:33:51.0111871Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T01:33:51.0175028Z Entering 'third_party/fbgemm/third_party/googletest' 2022-11-23T01:33:51.0232513Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T01:33:51.0307193Z Entering 'third_party/flatbuffers' 2022-11-23T01:33:51.0381472Z Entering 'third_party/fmt' 2022-11-23T01:33:51.0452833Z Entering 'third_party/foxi' 2022-11-23T01:33:51.0522536Z Entering 'third_party/gemmlowp/gemmlowp' 2022-11-23T01:33:51.0592474Z Entering 'third_party/gloo' 2022-11-23T01:33:51.0655533Z Entering 'third_party/googletest' 2022-11-23T01:33:51.0723100Z Entering 'third_party/ideep' 2022-11-23T01:33:51.0791926Z Entering 'third_party/ideep/mkl-dnn' 2022-11-23T01:33:51.0859503Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T01:33:51.0937321Z Entering 'third_party/ios-cmake' 2022-11-23T01:33:51.1004306Z Entering 'third_party/ittapi' 2022-11-23T01:33:51.1067311Z Entering 'third_party/kineto' 2022-11-23T01:33:51.1135101Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T01:33:51.1207323Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T01:33:51.1280131Z Entering 'third_party/nccl/nccl' 2022-11-23T01:33:51.1343769Z Entering 'third_party/neon2sse' 2022-11-23T01:33:51.1415043Z Entering 'third_party/nlohmann' 2022-11-23T01:33:51.1487174Z Entering 'third_party/onnx' 2022-11-23T01:33:51.1577248Z Entering 'third_party/onnx/third_party/benchmark' 2022-11-23T01:33:51.1641215Z Entering 'third_party/onnx/third_party/pybind11' 2022-11-23T01:33:51.1720159Z Entering 'third_party/onnx-tensorrt' 2022-11-23T01:33:51.1782201Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T01:33:51.1863431Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T01:33:51.1934277Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T01:33:51.2005640Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T01:33:51.2081411Z Entering 'third_party/pocketfft' 2022-11-23T01:33:51.2149732Z Entering 'third_party/protobuf' 2022-11-23T01:33:51.2214883Z Entering 'third_party/protobuf/third_party/benchmark' 2022-11-23T01:33:51.2280560Z Entering 'third_party/protobuf/third_party/googletest' 2022-11-23T01:33:51.2347402Z Entering 'third_party/psimd' 2022-11-23T01:33:51.2413476Z Entering 'third_party/pthreadpool' 2022-11-23T01:33:51.2483199Z Entering 'third_party/pybind11' 2022-11-23T01:33:51.2555452Z Entering 'third_party/python-enum' 2022-11-23T01:33:51.2617497Z Entering 'third_party/python-peachpy' 2022-11-23T01:33:51.2679689Z Entering 'third_party/python-six' 2022-11-23T01:33:51.2746900Z Entering 'third_party/sleef' 2022-11-23T01:33:51.2805032Z Entering 'third_party/tbb' 2022-11-23T01:33:51.2863411Z Entering 'third_party/tensorpipe' 2022-11-23T01:33:51.2932102Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-11-23T01:33:51.3001525Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-11-23T01:33:51.3061029Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-11-23T01:33:51.3125796Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T01:33:51.3190428Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T01:33:51.3263050Z Entering 'third_party/zstd' 2022-11-23T01:33:51.3346960Z ##[endgroup] 2022-11-23T01:33:51.3350442Z ##[group]Persisting credentials for submodules 2022-11-23T01:33:51.3359615Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || : 2022-11-23T01:33:51.3749531Z Entering 'android/libs/fbjni' 2022-11-23T01:33:51.3823329Z Entering 'third_party/FP16' 2022-11-23T01:33:51.3885852Z Entering 'third_party/FXdiv' 2022-11-23T01:33:51.3945087Z Entering 'third_party/NNPACK' 2022-11-23T01:33:51.4011909Z Entering 'third_party/QNNPACK' 2022-11-23T01:33:51.4081031Z Entering 'third_party/VulkanMemoryAllocator' 2022-11-23T01:33:51.4149468Z Entering 'third_party/XNNPACK' 2022-11-23T01:33:51.4219944Z Entering 'third_party/benchmark' 2022-11-23T01:33:51.4284130Z Entering 'third_party/cpuinfo' 2022-11-23T01:33:51.4340491Z Entering 'third_party/cub' 2022-11-23T01:33:51.4397241Z Entering 'third_party/cudnn_frontend' 2022-11-23T01:33:51.4480489Z Entering 'third_party/cutlass' 2022-11-23T01:33:51.4557066Z Entering 'third_party/eigen' 2022-11-23T01:33:51.4633106Z Entering 'third_party/fbgemm' 2022-11-23T01:33:51.4691838Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-11-23T01:33:51.4754647Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T01:33:51.4812650Z Entering 'third_party/fbgemm/third_party/googletest' 2022-11-23T01:33:51.4878464Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T01:33:51.4939583Z Entering 'third_party/flatbuffers' 2022-11-23T01:33:51.5003085Z Entering 'third_party/fmt' 2022-11-23T01:33:51.5072200Z Entering 'third_party/foxi' 2022-11-23T01:33:51.5140078Z Entering 'third_party/gemmlowp/gemmlowp' 2022-11-23T01:33:51.5209282Z Entering 'third_party/gloo' 2022-11-23T01:33:51.5276357Z Entering 'third_party/googletest' 2022-11-23T01:33:51.5345766Z Entering 'third_party/ideep' 2022-11-23T01:33:51.5407526Z Entering 'third_party/ideep/mkl-dnn' 2022-11-23T01:33:51.5476348Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T01:33:51.5558919Z Entering 'third_party/ios-cmake' 2022-11-23T01:33:51.5620654Z Entering 'third_party/ittapi' 2022-11-23T01:33:51.5681641Z Entering 'third_party/kineto' 2022-11-23T01:33:51.5752229Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T01:33:51.5821050Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T01:33:51.5877447Z Entering 'third_party/nccl/nccl' 2022-11-23T01:33:51.5938082Z Entering 'third_party/neon2sse' 2022-11-23T01:33:51.6004717Z Entering 'third_party/nlohmann' 2022-11-23T01:33:51.6077197Z Entering 'third_party/onnx' 2022-11-23T01:33:51.6173849Z Entering 'third_party/onnx/third_party/benchmark' 2022-11-23T01:33:51.6243791Z Entering 'third_party/onnx/third_party/pybind11' 2022-11-23T01:33:51.6321375Z Entering 'third_party/onnx-tensorrt' 2022-11-23T01:33:51.6389883Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T01:33:51.6463534Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T01:33:51.6524994Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T01:33:51.6582571Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T01:33:51.6653142Z Entering 'third_party/pocketfft' 2022-11-23T01:33:51.6715002Z Entering 'third_party/protobuf' 2022-11-23T01:33:51.6790866Z Entering 'third_party/protobuf/third_party/benchmark' 2022-11-23T01:33:51.6858106Z Entering 'third_party/protobuf/third_party/googletest' 2022-11-23T01:33:51.6925835Z Entering 'third_party/psimd' 2022-11-23T01:33:51.6995129Z Entering 'third_party/pthreadpool' 2022-11-23T01:33:51.7051447Z Entering 'third_party/pybind11' 2022-11-23T01:33:51.7119725Z Entering 'third_party/python-enum' 2022-11-23T01:33:51.7182283Z Entering 'third_party/python-peachpy' 2022-11-23T01:33:51.7250518Z Entering 'third_party/python-six' 2022-11-23T01:33:51.7320076Z Entering 'third_party/sleef' 2022-11-23T01:33:51.7390793Z Entering 'third_party/tbb' 2022-11-23T01:33:51.7461458Z Entering 'third_party/tensorpipe' 2022-11-23T01:33:51.7521233Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-11-23T01:33:51.7590655Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-11-23T01:33:51.7656449Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-11-23T01:33:51.7717628Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T01:33:51.7770779Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T01:33:51.7838856Z Entering 'third_party/zstd' 2022-11-23T01:33:51.7922071Z [command]/usr/bin/git submodule foreach --recursive git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url 2022-11-23T01:33:51.8324351Z Entering 'android/libs/fbjni' 2022-11-23T01:33:51.8377875Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2022-11-23T01:33:51.8410142Z Entering 'third_party/FP16' 2022-11-23T01:33:51.8460525Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2022-11-23T01:33:51.8487186Z Entering 'third_party/FXdiv' 2022-11-23T01:33:51.8549149Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2022-11-23T01:33:51.8581878Z Entering 'third_party/NNPACK' 2022-11-23T01:33:51.8632586Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2022-11-23T01:33:51.8668070Z Entering 'third_party/QNNPACK' 2022-11-23T01:33:51.8722753Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config remote.origin.url 2022-11-23T01:33:51.8753360Z Entering 'third_party/VulkanMemoryAllocator' 2022-11-23T01:33:51.8809871Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2022-11-23T01:33:51.8834691Z Entering 'third_party/XNNPACK' 2022-11-23T01:33:51.8892232Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2022-11-23T01:33:51.8929346Z Entering 'third_party/benchmark' 2022-11-23T01:33:51.8977741Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2022-11-23T01:33:51.9011623Z Entering 'third_party/cpuinfo' 2022-11-23T01:33:51.9062511Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2022-11-23T01:33:51.9097797Z Entering 'third_party/cub' 2022-11-23T01:33:51.9159317Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config remote.origin.url 2022-11-23T01:33:51.9195630Z Entering 'third_party/cudnn_frontend' 2022-11-23T01:33:51.9252741Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2022-11-23T01:33:51.9297787Z Entering 'third_party/cutlass' 2022-11-23T01:33:51.9359420Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2022-11-23T01:33:51.9400481Z Entering 'third_party/eigen' 2022-11-23T01:33:51.9463409Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2022-11-23T01:33:51.9503818Z Entering 'third_party/fbgemm' 2022-11-23T01:33:51.9559490Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2022-11-23T01:33:51.9593438Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-11-23T01:33:51.9651925Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2022-11-23T01:33:51.9684388Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T01:33:51.9747673Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2022-11-23T01:33:51.9772372Z Entering 'third_party/fbgemm/third_party/googletest' 2022-11-23T01:33:51.9826565Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2022-11-23T01:33:51.9851537Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T01:33:51.9911994Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2022-11-23T01:33:51.9950190Z Entering 'third_party/flatbuffers' 2022-11-23T01:33:52.0005989Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2022-11-23T01:33:52.0043212Z Entering 'third_party/fmt' 2022-11-23T01:33:52.0102454Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2022-11-23T01:33:52.0129046Z Entering 'third_party/foxi' 2022-11-23T01:33:52.0190603Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2022-11-23T01:33:52.0224593Z Entering 'third_party/gemmlowp/gemmlowp' 2022-11-23T01:33:52.0286897Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2022-11-23T01:33:52.0321155Z Entering 'third_party/gloo' 2022-11-23T01:33:52.0377927Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2022-11-23T01:33:52.0405210Z Entering 'third_party/googletest' 2022-11-23T01:33:52.0459842Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2022-11-23T01:33:52.0492557Z Entering 'third_party/ideep' 2022-11-23T01:33:52.0549367Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2022-11-23T01:33:52.0577599Z Entering 'third_party/ideep/mkl-dnn' 2022-11-23T01:33:52.0635623Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2022-11-23T01:33:52.0674865Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T01:33:52.0730681Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/modules/third_party/oneDNN/config remote.origin.url 2022-11-23T01:33:52.0764226Z Entering 'third_party/ios-cmake' 2022-11-23T01:33:52.0817165Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config remote.origin.url 2022-11-23T01:33:52.0850109Z Entering 'third_party/ittapi' 2022-11-23T01:33:52.0903385Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2022-11-23T01:33:52.0934841Z Entering 'third_party/kineto' 2022-11-23T01:33:52.0997264Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2022-11-23T01:33:52.1029729Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T01:33:52.1089620Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2022-11-23T01:33:52.1123123Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T01:33:52.1183668Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2022-11-23T01:33:52.1221977Z Entering 'third_party/nccl/nccl' 2022-11-23T01:33:52.1276915Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2022-11-23T01:33:52.1310619Z Entering 'third_party/neon2sse' 2022-11-23T01:33:52.1370897Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config remote.origin.url 2022-11-23T01:33:52.1403987Z Entering 'third_party/nlohmann' 2022-11-23T01:33:52.1450284Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2022-11-23T01:33:52.1484477Z Entering 'third_party/onnx' 2022-11-23T01:33:52.1543750Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2022-11-23T01:33:52.1603821Z Entering 'third_party/onnx/third_party/benchmark' 2022-11-23T01:33:52.1654906Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2022-11-23T01:33:52.1688209Z Entering 'third_party/onnx/third_party/pybind11' 2022-11-23T01:33:52.1746996Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2022-11-23T01:33:52.1776307Z Entering 'third_party/onnx-tensorrt' 2022-11-23T01:33:52.1835521Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config remote.origin.url 2022-11-23T01:33:52.1861433Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T01:33:52.1921718Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config remote.origin.url 2022-11-23T01:33:52.1966786Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T01:33:52.2023904Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2022-11-23T01:33:52.2055529Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T01:33:52.2111721Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2022-11-23T01:33:52.2142176Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T01:33:52.2194865Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2022-11-23T01:33:52.2241450Z Entering 'third_party/pocketfft' 2022-11-23T01:33:52.2300529Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2022-11-23T01:33:52.2333451Z Entering 'third_party/protobuf' 2022-11-23T01:33:52.2388819Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2022-11-23T01:33:52.2429245Z Entering 'third_party/protobuf/third_party/benchmark' 2022-11-23T01:33:52.2486154Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2022-11-23T01:33:52.2515464Z Entering 'third_party/protobuf/third_party/googletest' 2022-11-23T01:33:52.2571438Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2022-11-23T01:33:52.2609658Z Entering 'third_party/psimd' 2022-11-23T01:33:52.2670751Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2022-11-23T01:33:52.2701370Z Entering 'third_party/pthreadpool' 2022-11-23T01:33:52.2759034Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2022-11-23T01:33:52.2792808Z Entering 'third_party/pybind11' 2022-11-23T01:33:52.2846946Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2022-11-23T01:33:52.2880912Z Entering 'third_party/python-enum' 2022-11-23T01:33:52.2941799Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-enum/config remote.origin.url 2022-11-23T01:33:52.2971670Z Entering 'third_party/python-peachpy' 2022-11-23T01:33:52.3033139Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2022-11-23T01:33:52.3067426Z Entering 'third_party/python-six' 2022-11-23T01:33:52.3120239Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-six/config remote.origin.url 2022-11-23T01:33:52.3150297Z Entering 'third_party/sleef' 2022-11-23T01:33:52.3205844Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2022-11-23T01:33:52.3240499Z Entering 'third_party/tbb' 2022-11-23T01:33:52.3302194Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config remote.origin.url 2022-11-23T01:33:52.3339234Z Entering 'third_party/tensorpipe' 2022-11-23T01:33:52.3399281Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2022-11-23T01:33:52.3434677Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-11-23T01:33:52.3493790Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2022-11-23T01:33:52.3521342Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-11-23T01:33:52.3571653Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2022-11-23T01:33:52.3605271Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-11-23T01:33:52.3660224Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2022-11-23T01:33:52.3694032Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T01:33:52.3748942Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2022-11-23T01:33:52.3772365Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T01:33:52.3830984Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2022-11-23T01:33:52.3870451Z Entering 'third_party/zstd' 2022-11-23T01:33:52.3930633Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config remote.origin.url 2022-11-23T01:33:52.4253015Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2022-11-23T01:33:52.4653319Z Entering 'android/libs/fbjni' 2022-11-23T01:33:52.4717198Z Entering 'third_party/FP16' 2022-11-23T01:33:52.4787001Z Entering 'third_party/FXdiv' 2022-11-23T01:33:52.4842504Z Entering 'third_party/NNPACK' 2022-11-23T01:33:52.4900188Z Entering 'third_party/QNNPACK' 2022-11-23T01:33:52.4952987Z Entering 'third_party/VulkanMemoryAllocator' 2022-11-23T01:33:52.5001946Z Entering 'third_party/XNNPACK' 2022-11-23T01:33:52.5080524Z Entering 'third_party/benchmark' 2022-11-23T01:33:52.5131494Z Entering 'third_party/cpuinfo' 2022-11-23T01:33:52.5195613Z Entering 'third_party/cub' 2022-11-23T01:33:52.5259335Z Entering 'third_party/cudnn_frontend' 2022-11-23T01:33:52.5326048Z Entering 'third_party/cutlass' 2022-11-23T01:33:52.5407493Z Entering 'third_party/eigen' 2022-11-23T01:33:52.5460683Z Entering 'third_party/fbgemm' 2022-11-23T01:33:52.5528430Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-11-23T01:33:52.5587623Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T01:33:52.5656284Z Entering 'third_party/fbgemm/third_party/googletest' 2022-11-23T01:33:52.5729102Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T01:33:52.5803190Z Entering 'third_party/flatbuffers' 2022-11-23T01:33:52.5877875Z Entering 'third_party/fmt' 2022-11-23T01:33:52.5949023Z Entering 'third_party/foxi' 2022-11-23T01:33:52.6019515Z Entering 'third_party/gemmlowp/gemmlowp' 2022-11-23T01:33:52.6092091Z Entering 'third_party/gloo' 2022-11-23T01:33:52.6162423Z Entering 'third_party/googletest' 2022-11-23T01:33:52.6229243Z Entering 'third_party/ideep' 2022-11-23T01:33:52.6302091Z Entering 'third_party/ideep/mkl-dnn' 2022-11-23T01:33:52.6374598Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T01:33:52.6452784Z Entering 'third_party/ios-cmake' 2022-11-23T01:33:52.6512286Z Entering 'third_party/ittapi' 2022-11-23T01:33:52.6568307Z Entering 'third_party/kineto' 2022-11-23T01:33:52.6635856Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T01:33:52.6706762Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T01:33:52.6767510Z Entering 'third_party/nccl/nccl' 2022-11-23T01:33:52.6836860Z Entering 'third_party/neon2sse' 2022-11-23T01:33:52.6906896Z Entering 'third_party/nlohmann' 2022-11-23T01:33:52.6970907Z Entering 'third_party/onnx' 2022-11-23T01:33:52.7065029Z Entering 'third_party/onnx/third_party/benchmark' 2022-11-23T01:33:52.7140121Z Entering 'third_party/onnx/third_party/pybind11' 2022-11-23T01:33:52.7215108Z Entering 'third_party/onnx-tensorrt' 2022-11-23T01:33:52.7282268Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T01:33:52.7351886Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T01:33:52.7418278Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T01:33:52.7487155Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T01:33:52.7569157Z Entering 'third_party/pocketfft' 2022-11-23T01:33:52.7627980Z Entering 'third_party/protobuf' 2022-11-23T01:33:52.7707878Z Entering 'third_party/protobuf/third_party/benchmark' 2022-11-23T01:33:52.7771872Z Entering 'third_party/protobuf/third_party/googletest' 2022-11-23T01:33:52.7848441Z Entering 'third_party/psimd' 2022-11-23T01:33:52.7915360Z Entering 'third_party/pthreadpool' 2022-11-23T01:33:52.7986128Z Entering 'third_party/pybind11' 2022-11-23T01:33:52.8056837Z Entering 'third_party/python-enum' 2022-11-23T01:33:52.8121081Z Entering 'third_party/python-peachpy' 2022-11-23T01:33:52.8192543Z Entering 'third_party/python-six' 2022-11-23T01:33:52.8257345Z Entering 'third_party/sleef' 2022-11-23T01:33:52.8316102Z Entering 'third_party/tbb' 2022-11-23T01:33:52.8378175Z Entering 'third_party/tensorpipe' 2022-11-23T01:33:52.8450215Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-11-23T01:33:52.8520235Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-11-23T01:33:52.8585438Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-11-23T01:33:52.8655263Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T01:33:52.8722630Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T01:33:52.8801248Z Entering 'third_party/zstd' 2022-11-23T01:33:52.8896701Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2022-11-23T01:33:52.9311379Z Entering 'android/libs/fbjni' 2022-11-23T01:33:52.9374830Z Entering 'third_party/FP16' 2022-11-23T01:33:52.9445512Z Entering 'third_party/FXdiv' 2022-11-23T01:33:52.9515978Z Entering 'third_party/NNPACK' 2022-11-23T01:33:52.9589995Z Entering 'third_party/QNNPACK' 2022-11-23T01:33:52.9651001Z Entering 'third_party/VulkanMemoryAllocator' 2022-11-23T01:33:52.9717748Z Entering 'third_party/XNNPACK' 2022-11-23T01:33:52.9809291Z Entering 'third_party/benchmark' 2022-11-23T01:33:52.9881342Z Entering 'third_party/cpuinfo' 2022-11-23T01:33:52.9951377Z Entering 'third_party/cub' 2022-11-23T01:33:53.0017459Z Entering 'third_party/cudnn_frontend' 2022-11-23T01:33:53.0099584Z Entering 'third_party/cutlass' 2022-11-23T01:33:53.0173469Z Entering 'third_party/eigen' 2022-11-23T01:33:53.0242537Z Entering 'third_party/fbgemm' 2022-11-23T01:33:53.0310585Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-11-23T01:33:53.0369778Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T01:33:53.0440722Z Entering 'third_party/fbgemm/third_party/googletest' 2022-11-23T01:33:53.0500009Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T01:33:53.0565529Z Entering 'third_party/flatbuffers' 2022-11-23T01:33:53.0642561Z Entering 'third_party/fmt' 2022-11-23T01:33:53.0715006Z Entering 'third_party/foxi' 2022-11-23T01:33:53.0783772Z Entering 'third_party/gemmlowp/gemmlowp' 2022-11-23T01:33:53.0852186Z Entering 'third_party/gloo' 2022-11-23T01:33:53.0913749Z Entering 'third_party/googletest' 2022-11-23T01:33:53.0984221Z Entering 'third_party/ideep' 2022-11-23T01:33:53.1048374Z Entering 'third_party/ideep/mkl-dnn' 2022-11-23T01:33:53.1117935Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T01:33:53.1194636Z Entering 'third_party/ios-cmake' 2022-11-23T01:33:53.1258382Z Entering 'third_party/ittapi' 2022-11-23T01:33:53.1332576Z Entering 'third_party/kineto' 2022-11-23T01:33:53.1400673Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T01:33:53.1475791Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T01:33:53.1544119Z Entering 'third_party/nccl/nccl' 2022-11-23T01:33:53.1613469Z Entering 'third_party/neon2sse' 2022-11-23T01:33:53.1672956Z Entering 'third_party/nlohmann' 2022-11-23T01:33:53.1738576Z Entering 'third_party/onnx' 2022-11-23T01:33:53.1835811Z Entering 'third_party/onnx/third_party/benchmark' 2022-11-23T01:33:53.1910257Z Entering 'third_party/onnx/third_party/pybind11' 2022-11-23T01:33:53.1986562Z Entering 'third_party/onnx-tensorrt' 2022-11-23T01:33:53.2050452Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T01:33:53.2128175Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T01:33:53.2193919Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T01:33:53.2268112Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T01:33:53.2350248Z Entering 'third_party/pocketfft' 2022-11-23T01:33:53.2411967Z Entering 'third_party/protobuf' 2022-11-23T01:33:53.2490481Z Entering 'third_party/protobuf/third_party/benchmark' 2022-11-23T01:33:53.2554235Z Entering 'third_party/protobuf/third_party/googletest' 2022-11-23T01:33:53.2620083Z Entering 'third_party/psimd' 2022-11-23T01:33:53.2690191Z Entering 'third_party/pthreadpool' 2022-11-23T01:33:53.2757835Z Entering 'third_party/pybind11' 2022-11-23T01:33:53.2825877Z Entering 'third_party/python-enum' 2022-11-23T01:33:53.2890182Z Entering 'third_party/python-peachpy' 2022-11-23T01:33:53.2963734Z Entering 'third_party/python-six' 2022-11-23T01:33:53.3029652Z Entering 'third_party/sleef' 2022-11-23T01:33:53.3101092Z Entering 'third_party/tbb' 2022-11-23T01:33:53.3176413Z Entering 'third_party/tensorpipe' 2022-11-23T01:33:53.3234889Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-11-23T01:33:53.3303845Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-11-23T01:33:53.3358358Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-11-23T01:33:53.3429354Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T01:33:53.3492410Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T01:33:53.3566192Z Entering 'third_party/zstd' 2022-11-23T01:33:53.3652176Z ##[endgroup] 2022-11-23T01:33:53.3738377Z [command]/usr/bin/git log -1 --format='%H' 2022-11-23T01:33:53.3799126Z '1cfd3858ac54fe3883534309081631a0a892ba3f' 2022-11-23T01:33:53.4120228Z Prepare all required actions 2022-11-23T01:33:53.4164261Z ##[group]Run ./.github/actions/setup-rocm 2022-11-23T01:33:53.4164518Z env: 2022-11-23T01:33:53.4164748Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:33:53.4164989Z ##[endgroup] 2022-11-23T01:33:53.4184515Z ##[group]Run echo "DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock" >> "${GITHUB_ENV}" 2022-11-23T01:33:53.4184937Z echo "DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock" >> "${GITHUB_ENV}" 2022-11-23T01:33:53.4208003Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:33:53.4208286Z env: 2022-11-23T01:33:53.4208545Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:33:53.4208842Z ##[endgroup] 2022-11-23T01:33:53.4322474Z ##[group]Run cat /etc/os-release || true 2022-11-23T01:33:53.4323279Z cat /etc/os-release || true 2022-11-23T01:33:53.4324087Z cat /etc/apt/sources.list.d/rocm.list || true 2022-11-23T01:33:53.4324911Z cat /opt/rocm/.info/version || true 2022-11-23T01:33:53.4325647Z whoami 2022-11-23T01:33:53.4379841Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:33:53.4380116Z env: 2022-11-23T01:33:53.4380361Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:33:53.4380664Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:33:53.4380941Z ##[endgroup] 2022-11-23T01:33:53.4431703Z NAME="Ubuntu" 2022-11-23T01:33:53.4432453Z VERSION="18.04.5 LTS (Bionic Beaver)" 2022-11-23T01:33:53.4433112Z ID=ubuntu 2022-11-23T01:33:53.4433680Z ID_LIKE=debian 2022-11-23T01:33:53.4434812Z PRETTY_NAME="Ubuntu 18.04.5 LTS" 2022-11-23T01:33:53.4435486Z VERSION_ID="18.04" 2022-11-23T01:33:53.4436216Z HOME_URL="https://www.ubuntu.com/" 2022-11-23T01:33:53.4437143Z SUPPORT_URL="https://help.ubuntu.com/" 2022-11-23T01:33:53.4438888Z BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-11-23T01:33:53.4440751Z PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-11-23T01:33:53.4441862Z VERSION_CODENAME=bionic 2022-11-23T01:33:53.4442644Z UBUNTU_CODENAME=bionic 2022-11-23T01:33:53.4443568Z deb [arch=amd64] http://repo.radeon.com/rocm/apt/4.2 xenial main 2022-11-23T01:33:53.4449700Z 4.2.0-21 2022-11-23T01:33:53.4467179Z pytorchci 2022-11-23T01:33:53.4513008Z ##[group]Run rocm-smi 2022-11-23T01:33:53.4513696Z rocm-smi 2022-11-23T01:33:53.4566402Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:33:53.4567139Z env: 2022-11-23T01:33:53.4567903Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:33:53.4568716Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:33:53.4569437Z ##[endgroup] 2022-11-23T01:33:53.5780753Z 2022-11-23T01:33:53.5781011Z 2022-11-23T01:33:53.5781653Z ======================= ROCm System Management Interface ======================= 2022-11-23T01:33:53.5782384Z ================================= Concise Info ================================= 2022-11-23T01:33:53.5783095Z GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 2022-11-23T01:33:53.5783788Z 0 36.0c 20.0W 930Mhz 350Mhz 255% auto 225.0W 0% 0% 2022-11-23T01:33:53.5784401Z 1 35.0c 17.0W 930Mhz 350Mhz 255% auto 225.0W 0% 0% 2022-11-23T01:33:53.5785003Z 2 32.0c 15.0W 930Mhz 350Mhz 255% auto 225.0W 0% 0% 2022-11-23T01:33:53.5785609Z 3 34.0c 15.0W 930Mhz 350Mhz 255% auto 225.0W 0% 0% 2022-11-23T01:33:53.5786187Z ================================================================================ 2022-11-23T01:33:53.5786783Z ============================= End of ROCm SMI Log ============================== 2022-11-23T01:33:53.5895175Z ##[group]Run rocminfo 2022-11-23T01:33:53.5895845Z rocminfo 2022-11-23T01:33:53.5935501Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:33:53.5936032Z env: 2022-11-23T01:33:53.5936458Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:33:53.5937008Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:33:53.5937497Z ##[endgroup] 2022-11-23T01:33:53.6943766Z ROCk module is loaded 2022-11-23T01:33:53.6944521Z ===================== 2022-11-23T01:33:53.6945177Z HSA System Attributes 2022-11-23T01:33:53.6945833Z ===================== 2022-11-23T01:33:53.6946460Z Runtime Version: 1.1 2022-11-23T01:33:53.6947181Z System Timestamp Freq.: 1000.000000MHz 2022-11-23T01:33:53.6948111Z Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count) 2022-11-23T01:33:53.6949065Z Machine Model: LARGE 2022-11-23T01:33:53.6949917Z System Endianness: LITTLE 2022-11-23T01:33:53.6950509Z 2022-11-23T01:33:53.6950751Z ========== 2022-11-23T01:33:53.6951382Z HSA Agents 2022-11-23T01:33:53.6952016Z ========== 2022-11-23T01:33:53.6952616Z ******* 2022-11-23T01:33:53.6953219Z Agent 1 2022-11-23T01:33:53.6953817Z ******* 2022-11-23T01:33:53.6954854Z Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:33:53.6955930Z Uuid: CPU-XX 2022-11-23T01:33:53.6956993Z Marketing Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:33:53.6957847Z Vendor Name: CPU 2022-11-23T01:33:53.6958658Z Feature: None specified 2022-11-23T01:33:53.6959491Z Profile: FULL_PROFILE 2022-11-23T01:33:53.6960538Z Float Round Mode: NEAR 2022-11-23T01:33:53.6961344Z Max Queue Number: 0(0x0) 2022-11-23T01:33:53.6962135Z Queue Min Size: 0(0x0) 2022-11-23T01:33:53.6962912Z Queue Max Size: 0(0x0) 2022-11-23T01:33:53.6963703Z Queue Type: MULTI 2022-11-23T01:33:53.6964472Z Node: 0 2022-11-23T01:33:53.6965235Z Device Type: CPU 2022-11-23T01:33:53.6965935Z Cache Info: 2022-11-23T01:33:53.6966653Z L1: 32768(0x8000) KB 2022-11-23T01:33:53.6967402Z Chip ID: 0(0x0) 2022-11-23T01:33:53.6968568Z Cacheline Size: 64(0x40) 2022-11-23T01:33:53.6969363Z Max Clock Freq. (MHz): 2200 2022-11-23T01:33:53.6970144Z BDFID: 0 2022-11-23T01:33:53.6970907Z Internal Node ID: 0 2022-11-23T01:33:53.6971682Z Compute Unit: 16 2022-11-23T01:33:53.6972438Z SIMDs per CU: 0 2022-11-23T01:33:53.6973221Z Shader Engines: 0 2022-11-23T01:33:53.6974029Z Shader Arrs. per Eng.: 0 2022-11-23T01:33:53.6974852Z WatchPts on Addr. Ranges:1 2022-11-23T01:33:53.6975588Z Features: None 2022-11-23T01:33:53.6976246Z Pool Info: 2022-11-23T01:33:53.6976885Z Pool 1 2022-11-23T01:33:53.6977679Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2022-11-23T01:33:53.6978532Z Size: 131954832(0x7dd7890) KB 2022-11-23T01:33:53.6979347Z Allocatable: TRUE 2022-11-23T01:33:53.6980160Z Alloc Granule: 4KB 2022-11-23T01:33:53.6980969Z Alloc Alignment: 4KB 2022-11-23T01:33:53.6981794Z Accessible by all: TRUE 2022-11-23T01:33:53.6982501Z Pool 2 2022-11-23T01:33:53.6983468Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:33:53.6984315Z Size: 131954832(0x7dd7890) KB 2022-11-23T01:33:53.6985125Z Allocatable: TRUE 2022-11-23T01:33:53.6985929Z Alloc Granule: 4KB 2022-11-23T01:33:53.6986737Z Alloc Alignment: 4KB 2022-11-23T01:33:53.6987538Z Accessible by all: TRUE 2022-11-23T01:33:53.6988266Z ISA Info: 2022-11-23T01:33:53.6988952Z N/A 2022-11-23T01:33:53.6989544Z ******* 2022-11-23T01:33:53.6990149Z Agent 2 2022-11-23T01:33:53.6990752Z ******* 2022-11-23T01:33:53.6991762Z Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:33:53.6992803Z Uuid: CPU-XX 2022-11-23T01:33:53.6993875Z Marketing Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:33:53.6994704Z Vendor Name: CPU 2022-11-23T01:33:53.6995493Z Feature: None specified 2022-11-23T01:33:53.6996312Z Profile: FULL_PROFILE 2022-11-23T01:33:53.6997118Z Float Round Mode: NEAR 2022-11-23T01:33:53.6997916Z Max Queue Number: 0(0x0) 2022-11-23T01:33:53.6998869Z Queue Min Size: 0(0x0) 2022-11-23T01:33:53.6999640Z Queue Max Size: 0(0x0) 2022-11-23T01:33:53.7000431Z Queue Type: MULTI 2022-11-23T01:33:53.7001180Z Node: 1 2022-11-23T01:33:53.7002259Z Device Type: CPU 2022-11-23T01:33:53.7002969Z Cache Info: 2022-11-23T01:33:53.7003688Z L1: 32768(0x8000) KB 2022-11-23T01:33:53.7004449Z Chip ID: 0(0x0) 2022-11-23T01:33:53.7005203Z Cacheline Size: 64(0x40) 2022-11-23T01:33:53.7005996Z Max Clock Freq. (MHz): 2200 2022-11-23T01:33:53.7006763Z BDFID: 0 2022-11-23T01:33:53.7007521Z Internal Node ID: 1 2022-11-23T01:33:53.7008415Z Compute Unit: 16 2022-11-23T01:33:53.7009194Z SIMDs per CU: 0 2022-11-23T01:33:53.7009967Z Shader Engines: 0 2022-11-23T01:33:53.7010770Z Shader Arrs. per Eng.: 0 2022-11-23T01:33:53.7011596Z WatchPts on Addr. Ranges:1 2022-11-23T01:33:53.7012352Z Features: None 2022-11-23T01:33:53.7013016Z Pool Info: 2022-11-23T01:33:53.7013672Z Pool 1 2022-11-23T01:33:53.7014447Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2022-11-23T01:33:53.7015306Z Size: 132087932(0x7df807c) KB 2022-11-23T01:33:53.7016118Z Allocatable: TRUE 2022-11-23T01:33:53.7016931Z Alloc Granule: 4KB 2022-11-23T01:33:53.7017751Z Alloc Alignment: 4KB 2022-11-23T01:33:53.7018581Z Accessible by all: TRUE 2022-11-23T01:33:53.7019291Z Pool 2 2022-11-23T01:33:53.7020073Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:33:53.7020921Z Size: 132087932(0x7df807c) KB 2022-11-23T01:33:53.7021878Z Allocatable: TRUE 2022-11-23T01:33:53.7022703Z Alloc Granule: 4KB 2022-11-23T01:33:53.7023516Z Alloc Alignment: 4KB 2022-11-23T01:33:53.7024346Z Accessible by all: TRUE 2022-11-23T01:33:53.7025058Z ISA Info: 2022-11-23T01:33:53.7025698Z N/A 2022-11-23T01:33:53.7026308Z ******* 2022-11-23T01:33:53.7026928Z Agent 3 2022-11-23T01:33:53.7027541Z ******* 2022-11-23T01:33:53.7028543Z Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:33:53.7029591Z Uuid: CPU-XX 2022-11-23T01:33:53.7030669Z Marketing Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:33:53.7031517Z Vendor Name: CPU 2022-11-23T01:33:53.7032334Z Feature: None specified 2022-11-23T01:33:53.7033158Z Profile: FULL_PROFILE 2022-11-23T01:33:53.7033979Z Float Round Mode: NEAR 2022-11-23T01:33:53.7034773Z Max Queue Number: 0(0x0) 2022-11-23T01:33:53.7035565Z Queue Min Size: 0(0x0) 2022-11-23T01:33:53.7036359Z Queue Max Size: 0(0x0) 2022-11-23T01:33:53.7037311Z Queue Type: MULTI 2022-11-23T01:33:53.7038080Z Node: 2 2022-11-23T01:33:53.7038837Z Device Type: CPU 2022-11-23T01:33:53.7039525Z Cache Info: 2022-11-23T01:33:53.7040250Z L1: 32768(0x8000) KB 2022-11-23T01:33:53.7041123Z Chip ID: 0(0x0) 2022-11-23T01:33:53.7041902Z Cacheline Size: 64(0x40) 2022-11-23T01:33:53.7042703Z Max Clock Freq. (MHz): 2200 2022-11-23T01:33:53.7043471Z BDFID: 0 2022-11-23T01:33:53.7044242Z Internal Node ID: 2 2022-11-23T01:33:53.7045071Z Compute Unit: 16 2022-11-23T01:33:53.7045884Z SIMDs per CU: 0 2022-11-23T01:33:53.7046664Z Shader Engines: 0 2022-11-23T01:33:53.7047502Z Shader Arrs. per Eng.: 0 2022-11-23T01:33:53.7048451Z WatchPts on Addr. Ranges:1 2022-11-23T01:33:53.7049201Z Features: None 2022-11-23T01:33:53.7049891Z Pool Info: 2022-11-23T01:33:53.7050579Z Pool 1 2022-11-23T01:33:53.7051372Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2022-11-23T01:33:53.7052238Z Size: 132112788(0x7dfe194) KB 2022-11-23T01:33:53.7053078Z Allocatable: TRUE 2022-11-23T01:33:53.7053913Z Alloc Granule: 4KB 2022-11-23T01:33:53.7054729Z Alloc Alignment: 4KB 2022-11-23T01:33:53.7055587Z Accessible by all: TRUE 2022-11-23T01:33:53.7056335Z Pool 2 2022-11-23T01:33:53.7057121Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:33:53.7057975Z Size: 132112788(0x7dfe194) KB 2022-11-23T01:33:53.7058786Z Allocatable: TRUE 2022-11-23T01:33:53.7059754Z Alloc Granule: 4KB 2022-11-23T01:33:53.7060603Z Alloc Alignment: 4KB 2022-11-23T01:33:53.7061444Z Accessible by all: TRUE 2022-11-23T01:33:53.7062173Z ISA Info: 2022-11-23T01:33:53.7062841Z N/A 2022-11-23T01:33:53.7063475Z ******* 2022-11-23T01:33:53.7064065Z Agent 4 2022-11-23T01:33:53.7064697Z ******* 2022-11-23T01:33:53.7065724Z Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:33:53.7066801Z Uuid: CPU-XX 2022-11-23T01:33:53.7067909Z Marketing Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:33:53.7068764Z Vendor Name: CPU 2022-11-23T01:33:53.7069577Z Feature: None specified 2022-11-23T01:33:53.7070417Z Profile: FULL_PROFILE 2022-11-23T01:33:53.7071246Z Float Round Mode: NEAR 2022-11-23T01:33:53.7072060Z Max Queue Number: 0(0x0) 2022-11-23T01:33:53.7072861Z Queue Min Size: 0(0x0) 2022-11-23T01:33:53.7073650Z Queue Max Size: 0(0x0) 2022-11-23T01:33:53.7074435Z Queue Type: MULTI 2022-11-23T01:33:53.7075201Z Node: 3 2022-11-23T01:33:53.7076140Z Device Type: CPU 2022-11-23T01:33:53.7076858Z Cache Info: 2022-11-23T01:33:53.7077617Z L1: 32768(0x8000) KB 2022-11-23T01:33:53.7078383Z Chip ID: 0(0x0) 2022-11-23T01:33:53.7079142Z Cacheline Size: 64(0x40) 2022-11-23T01:33:53.7079954Z Max Clock Freq. (MHz): 2200 2022-11-23T01:33:53.7080726Z BDFID: 0 2022-11-23T01:33:53.7081519Z Internal Node ID: 3 2022-11-23T01:33:53.7082307Z Compute Unit: 16 2022-11-23T01:33:53.7083089Z SIMDs per CU: 0 2022-11-23T01:33:53.7083852Z Shader Engines: 0 2022-11-23T01:33:53.7084675Z Shader Arrs. per Eng.: 0 2022-11-23T01:33:53.7085515Z WatchPts on Addr. Ranges:1 2022-11-23T01:33:53.7086253Z Features: None 2022-11-23T01:33:53.7086924Z Pool Info: 2022-11-23T01:33:53.7087587Z Pool 1 2022-11-23T01:33:53.7088672Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2022-11-23T01:33:53.7089549Z Size: 132111260(0x7dfdb9c) KB 2022-11-23T01:33:53.7090365Z Allocatable: TRUE 2022-11-23T01:33:53.7091215Z Alloc Granule: 4KB 2022-11-23T01:33:53.7092032Z Alloc Alignment: 4KB 2022-11-23T01:33:53.7092902Z Accessible by all: TRUE 2022-11-23T01:33:53.7093661Z Pool 2 2022-11-23T01:33:53.7094467Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:33:53.7095357Z Size: 132111260(0x7dfdb9c) KB 2022-11-23T01:33:53.7096165Z Allocatable: TRUE 2022-11-23T01:33:53.7097014Z Alloc Granule: 4KB 2022-11-23T01:33:53.7097928Z Alloc Alignment: 4KB 2022-11-23T01:33:53.7099325Z Accessible by all: TRUE 2022-11-23T01:33:53.7100682Z ISA Info: 2022-11-23T01:33:53.7101965Z N/A 2022-11-23T01:33:53.7102861Z ******* 2022-11-23T01:33:53.7103749Z Agent 5 2022-11-23T01:33:53.7104069Z ******* 2022-11-23T01:33:53.7104385Z Name: gfx906 2022-11-23T01:33:53.7104842Z Uuid: GPU-621e518172da5ee8 2022-11-23T01:33:53.7105210Z Marketing Name: Vega 20 2022-11-23T01:33:53.7105570Z Vendor Name: AMD 2022-11-23T01:33:53.7105972Z Feature: KERNEL_DISPATCH 2022-11-23T01:33:53.7106360Z Profile: BASE_PROFILE 2022-11-23T01:33:53.7106735Z Float Round Mode: NEAR 2022-11-23T01:33:53.7107101Z Max Queue Number: 128(0x80) 2022-11-23T01:33:53.7107462Z Queue Min Size: 4096(0x1000) 2022-11-23T01:33:53.7107819Z Queue Max Size: 131072(0x20000) 2022-11-23T01:33:53.7108170Z Queue Type: MULTI 2022-11-23T01:33:53.7108514Z Node: 4 2022-11-23T01:33:53.7108854Z Device Type: GPU 2022-11-23T01:33:53.7109303Z Cache Info: 2022-11-23T01:33:53.7109631Z L1: 16(0x10) KB 2022-11-23T01:33:53.7109977Z Chip ID: 26273(0x66a1) 2022-11-23T01:33:53.7110322Z Cacheline Size: 64(0x40) 2022-11-23T01:33:53.7110681Z Max Clock Freq. (MHz): 1725 2022-11-23T01:33:53.7111032Z BDFID: 8960 2022-11-23T01:33:53.7111384Z Internal Node ID: 4 2022-11-23T01:33:53.7111736Z Compute Unit: 60 2022-11-23T01:33:53.7112083Z SIMDs per CU: 4 2022-11-23T01:33:53.7112423Z Shader Engines: 4 2022-11-23T01:33:53.7112790Z Shader Arrs. per Eng.: 1 2022-11-23T01:33:53.7113168Z WatchPts on Addr. Ranges:4 2022-11-23T01:33:53.7113525Z Features: KERNEL_DISPATCH 2022-11-23T01:33:53.7113890Z Fast F16 Operation: FALSE 2022-11-23T01:33:53.7114263Z Wavefront Size: 64(0x40) 2022-11-23T01:33:53.7114618Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:33:53.7114979Z Workgroup Max Size per Dimension: 2022-11-23T01:33:53.7115336Z x 1024(0x400) 2022-11-23T01:33:53.7115669Z y 1024(0x400) 2022-11-23T01:33:53.7116005Z z 1024(0x400) 2022-11-23T01:33:53.7116360Z Max Waves Per CU: 40(0x28) 2022-11-23T01:33:53.7116846Z Max Work-item Per CU: 2560(0xa00) 2022-11-23T01:33:53.7117223Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:33:53.7117570Z Grid Max Size per Dimension: 2022-11-23T01:33:53.7117914Z x 4294967295(0xffffffff) 2022-11-23T01:33:53.7118268Z y 4294967295(0xffffffff) 2022-11-23T01:33:53.7118616Z z 4294967295(0xffffffff) 2022-11-23T01:33:53.7118988Z Max fbarriers/Workgrp: 32 2022-11-23T01:33:53.7119306Z Pool Info: 2022-11-23T01:33:53.7119670Z Pool 1 2022-11-23T01:33:53.7120032Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:33:53.7120419Z Size: 16760832(0xffc000) KB 2022-11-23T01:33:53.7120783Z Allocatable: TRUE 2022-11-23T01:33:53.7121147Z Alloc Granule: 4KB 2022-11-23T01:33:53.7121503Z Alloc Alignment: 4KB 2022-11-23T01:33:53.7121885Z Accessible by all: FALSE 2022-11-23T01:33:53.7122218Z Pool 2 2022-11-23T01:33:53.7122551Z Segment: GROUP 2022-11-23T01:33:53.7122900Z Size: 64(0x40) KB 2022-11-23T01:33:53.7123272Z Allocatable: FALSE 2022-11-23T01:33:53.7123589Z Alloc Granule: 0KB 2022-11-23T01:33:53.7123894Z Alloc Alignment: 0KB 2022-11-23T01:33:53.7124207Z Accessible by all: FALSE 2022-11-23T01:33:53.7124482Z ISA Info: 2022-11-23T01:33:53.7124727Z ISA 1 2022-11-23T01:33:53.7125136Z Name: amdgcn-amd-amdhsa--gfx906:sramecc+:xnack- 2022-11-23T01:33:53.7125489Z Machine Models: HSA_MACHINE_MODEL_LARGE 2022-11-23T01:33:53.7125882Z Profiles: HSA_PROFILE_BASE 2022-11-23T01:33:53.7126205Z Default Rounding Mode: NEAR 2022-11-23T01:33:53.7126525Z Default Rounding Mode: NEAR 2022-11-23T01:33:53.7126833Z Fast f16: TRUE 2022-11-23T01:33:53.7127140Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:33:53.7127440Z Workgroup Max Size per Dimension: 2022-11-23T01:33:53.7127800Z x 1024(0x400) 2022-11-23T01:33:53.7128089Z y 1024(0x400) 2022-11-23T01:33:53.7128372Z z 1024(0x400) 2022-11-23T01:33:53.7128718Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:33:53.7129058Z Grid Max Size per Dimension: 2022-11-23T01:33:53.7129417Z x 4294967295(0xffffffff) 2022-11-23T01:33:53.7129761Z y 4294967295(0xffffffff) 2022-11-23T01:33:53.7130119Z z 4294967295(0xffffffff) 2022-11-23T01:33:53.7130484Z FBarrier Max Size: 32 2022-11-23T01:33:53.7130807Z ******* 2022-11-23T01:33:53.7131082Z Agent 6 2022-11-23T01:33:53.7131363Z ******* 2022-11-23T01:33:53.7131671Z Name: gfx906 2022-11-23T01:33:53.7132140Z Uuid: GPU-4410584172da5ebc 2022-11-23T01:33:53.7132504Z Marketing Name: Vega 20 2022-11-23T01:33:53.7132867Z Vendor Name: AMD 2022-11-23T01:33:53.7133229Z Feature: KERNEL_DISPATCH 2022-11-23T01:33:53.7133608Z Profile: BASE_PROFILE 2022-11-23T01:33:53.7133965Z Float Round Mode: NEAR 2022-11-23T01:33:53.7134336Z Max Queue Number: 128(0x80) 2022-11-23T01:33:53.7134698Z Queue Min Size: 4096(0x1000) 2022-11-23T01:33:53.7135050Z Queue Max Size: 131072(0x20000) 2022-11-23T01:33:53.7135496Z Queue Type: MULTI 2022-11-23T01:33:53.7135851Z Node: 5 2022-11-23T01:33:53.7136187Z Device Type: GPU 2022-11-23T01:33:53.7136505Z Cache Info: 2022-11-23T01:33:53.7136834Z L1: 16(0x10) KB 2022-11-23T01:33:53.7137184Z Chip ID: 26273(0x66a1) 2022-11-23T01:33:53.7137541Z Cacheline Size: 64(0x40) 2022-11-23T01:33:53.7137904Z Max Clock Freq. (MHz): 1725 2022-11-23T01:33:53.7138250Z BDFID: 9728 2022-11-23T01:33:53.7138587Z Internal Node ID: 5 2022-11-23T01:33:53.7138939Z Compute Unit: 60 2022-11-23T01:33:53.7139297Z SIMDs per CU: 4 2022-11-23T01:33:53.7139653Z Shader Engines: 4 2022-11-23T01:33:53.7140013Z Shader Arrs. per Eng.: 1 2022-11-23T01:33:53.7140384Z WatchPts on Addr. Ranges:4 2022-11-23T01:33:53.7140742Z Features: KERNEL_DISPATCH 2022-11-23T01:33:53.7141107Z Fast F16 Operation: FALSE 2022-11-23T01:33:53.7141467Z Wavefront Size: 64(0x40) 2022-11-23T01:33:53.7141905Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:33:53.7142257Z Workgroup Max Size per Dimension: 2022-11-23T01:33:53.7142609Z x 1024(0x400) 2022-11-23T01:33:53.7142933Z y 1024(0x400) 2022-11-23T01:33:53.7143263Z z 1024(0x400) 2022-11-23T01:33:53.7143627Z Max Waves Per CU: 40(0x28) 2022-11-23T01:33:53.7144073Z Max Work-item Per CU: 2560(0xa00) 2022-11-23T01:33:53.7144387Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:33:53.7144697Z Grid Max Size per Dimension: 2022-11-23T01:33:53.7144978Z x 4294967295(0xffffffff) 2022-11-23T01:33:53.7145275Z y 4294967295(0xffffffff) 2022-11-23T01:33:53.7145569Z z 4294967295(0xffffffff) 2022-11-23T01:33:53.7145883Z Max fbarriers/Workgrp: 32 2022-11-23T01:33:53.7146155Z Pool Info: 2022-11-23T01:33:53.7146405Z Pool 1 2022-11-23T01:33:53.7146694Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:33:53.7147017Z Size: 16760832(0xffc000) KB 2022-11-23T01:33:53.7147326Z Allocatable: TRUE 2022-11-23T01:33:53.7147635Z Alloc Granule: 4KB 2022-11-23T01:33:53.7147946Z Alloc Alignment: 4KB 2022-11-23T01:33:53.7148261Z Accessible by all: FALSE 2022-11-23T01:33:53.7148542Z Pool 2 2022-11-23T01:33:53.7148812Z Segment: GROUP 2022-11-23T01:33:53.7149109Z Size: 64(0x40) KB 2022-11-23T01:33:53.7149415Z Allocatable: FALSE 2022-11-23T01:33:53.7149727Z Alloc Granule: 0KB 2022-11-23T01:33:53.7150041Z Alloc Alignment: 0KB 2022-11-23T01:33:53.7150356Z Accessible by all: FALSE 2022-11-23T01:33:53.7150626Z ISA Info: 2022-11-23T01:33:53.7150928Z ISA 1 2022-11-23T01:33:53.7151344Z Name: amdgcn-amd-amdhsa--gfx906:sramecc+:xnack- 2022-11-23T01:33:53.7151709Z Machine Models: HSA_MACHINE_MODEL_LARGE 2022-11-23T01:33:53.7152043Z Profiles: HSA_PROFILE_BASE 2022-11-23T01:33:53.7152369Z Default Rounding Mode: NEAR 2022-11-23T01:33:53.7152682Z Default Rounding Mode: NEAR 2022-11-23T01:33:53.7152997Z Fast f16: TRUE 2022-11-23T01:33:53.7153304Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:33:53.7153607Z Workgroup Max Size per Dimension: 2022-11-23T01:33:53.7153904Z x 1024(0x400) 2022-11-23T01:33:53.7154189Z y 1024(0x400) 2022-11-23T01:33:53.7154479Z z 1024(0x400) 2022-11-23T01:33:53.7154771Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:33:53.7155067Z Grid Max Size per Dimension: 2022-11-23T01:33:53.7155360Z x 4294967295(0xffffffff) 2022-11-23T01:33:53.7155668Z y 4294967295(0xffffffff) 2022-11-23T01:33:53.7155970Z z 4294967295(0xffffffff) 2022-11-23T01:33:53.7156327Z FBarrier Max Size: 32 2022-11-23T01:33:53.7156583Z ******* 2022-11-23T01:33:53.7156815Z Agent 7 2022-11-23T01:33:53.7157048Z ******* 2022-11-23T01:33:53.7157313Z Name: gfx906 2022-11-23T01:33:53.7157704Z Uuid: GPU-61f2486172da5ee8 2022-11-23T01:33:53.7158011Z Marketing Name: Vega 20 2022-11-23T01:33:53.7158306Z Vendor Name: AMD 2022-11-23T01:33:53.7158617Z Feature: KERNEL_DISPATCH 2022-11-23T01:33:53.7158932Z Profile: BASE_PROFILE 2022-11-23T01:33:53.7159243Z Float Round Mode: NEAR 2022-11-23T01:33:53.7159546Z Max Queue Number: 128(0x80) 2022-11-23T01:33:53.7159847Z Queue Min Size: 4096(0x1000) 2022-11-23T01:33:53.7160145Z Queue Max Size: 131072(0x20000) 2022-11-23T01:33:53.7160445Z Queue Type: MULTI 2022-11-23T01:33:53.7160733Z Node: 6 2022-11-23T01:33:53.7161021Z Device Type: GPU 2022-11-23T01:33:53.7161291Z Cache Info: 2022-11-23T01:33:53.7161569Z L1: 16(0x10) KB 2022-11-23T01:33:53.7161849Z Chip ID: 26273(0x66a1) 2022-11-23T01:33:53.7162145Z Cacheline Size: 64(0x40) 2022-11-23T01:33:53.7162446Z Max Clock Freq. (MHz): 1725 2022-11-23T01:33:53.7162736Z BDFID: 25344 2022-11-23T01:33:53.7163027Z Internal Node ID: 6 2022-11-23T01:33:53.7163326Z Compute Unit: 60 2022-11-23T01:33:53.7163616Z SIMDs per CU: 4 2022-11-23T01:33:53.7163899Z Shader Engines: 4 2022-11-23T01:33:53.7164203Z Shader Arrs. per Eng.: 1 2022-11-23T01:33:53.7164519Z WatchPts on Addr. Ranges:4 2022-11-23T01:33:53.7164873Z Features: KERNEL_DISPATCH 2022-11-23T01:33:53.7165191Z Fast F16 Operation: FALSE 2022-11-23T01:33:53.7165497Z Wavefront Size: 64(0x40) 2022-11-23T01:33:53.7165793Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:33:53.7166085Z Workgroup Max Size per Dimension: 2022-11-23T01:33:53.7166380Z x 1024(0x400) 2022-11-23T01:33:53.7166659Z y 1024(0x400) 2022-11-23T01:33:53.7166941Z z 1024(0x400) 2022-11-23T01:33:53.7167237Z Max Waves Per CU: 40(0x28) 2022-11-23T01:33:53.7167627Z Max Work-item Per CU: 2560(0xa00) 2022-11-23T01:33:53.7168030Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:33:53.7168319Z Grid Max Size per Dimension: 2022-11-23T01:33:53.7168650Z x 4294967295(0xffffffff) 2022-11-23T01:33:53.7169000Z y 4294967295(0xffffffff) 2022-11-23T01:33:53.7169342Z z 4294967295(0xffffffff) 2022-11-23T01:33:53.7169686Z Max fbarriers/Workgrp: 32 2022-11-23T01:33:53.7170015Z Pool Info: 2022-11-23T01:33:53.7170312Z Pool 1 2022-11-23T01:33:53.7170666Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:33:53.7171143Z Size: 16760832(0xffc000) KB 2022-11-23T01:33:53.7171508Z Allocatable: TRUE 2022-11-23T01:33:53.7171878Z Alloc Granule: 4KB 2022-11-23T01:33:53.7172248Z Alloc Alignment: 4KB 2022-11-23T01:33:53.7172628Z Accessible by all: FALSE 2022-11-23T01:33:53.7172960Z Pool 2 2022-11-23T01:33:53.7173295Z Segment: GROUP 2022-11-23T01:33:53.7173652Z Size: 64(0x40) KB 2022-11-23T01:33:53.7174013Z Allocatable: FALSE 2022-11-23T01:33:53.7174373Z Alloc Granule: 0KB 2022-11-23T01:33:53.7174737Z Alloc Alignment: 0KB 2022-11-23T01:33:53.7175113Z Accessible by all: FALSE 2022-11-23T01:33:53.7175447Z ISA Info: 2022-11-23T01:33:53.7175745Z ISA 1 2022-11-23T01:33:53.7176251Z Name: amdgcn-amd-amdhsa--gfx906:sramecc+:xnack- 2022-11-23T01:33:53.7176672Z Machine Models: HSA_MACHINE_MODEL_LARGE 2022-11-23T01:33:53.7177079Z Profiles: HSA_PROFILE_BASE 2022-11-23T01:33:53.7177471Z Default Rounding Mode: NEAR 2022-11-23T01:33:53.7177859Z Default Rounding Mode: NEAR 2022-11-23T01:33:53.7178225Z Fast f16: TRUE 2022-11-23T01:33:53.7178588Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:33:53.7178936Z Workgroup Max Size per Dimension: 2022-11-23T01:33:53.7179294Z x 1024(0x400) 2022-11-23T01:33:53.7179642Z y 1024(0x400) 2022-11-23T01:33:53.7179982Z z 1024(0x400) 2022-11-23T01:33:53.7180348Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:33:53.7180700Z Grid Max Size per Dimension: 2022-11-23T01:33:53.7181109Z x 4294967295(0xffffffff) 2022-11-23T01:33:53.7181474Z y 4294967295(0xffffffff) 2022-11-23T01:33:53.7181847Z z 4294967295(0xffffffff) 2022-11-23T01:33:53.7182220Z FBarrier Max Size: 32 2022-11-23T01:33:53.7182533Z ******* 2022-11-23T01:33:53.7182812Z Agent 8 2022-11-23T01:33:53.7183073Z ******* 2022-11-23T01:33:53.7183395Z Name: gfx906 2022-11-23T01:33:53.7183842Z Uuid: GPU-4a22612172fd5d11 2022-11-23T01:33:53.7184146Z Marketing Name: Vega 20 2022-11-23T01:33:53.7184524Z Vendor Name: AMD 2022-11-23T01:33:53.7184954Z Feature: KERNEL_DISPATCH 2022-11-23T01:33:53.7185300Z Profile: BASE_PROFILE 2022-11-23T01:33:53.7185643Z Float Round Mode: NEAR 2022-11-23T01:33:53.7185954Z Max Queue Number: 128(0x80) 2022-11-23T01:33:53.7186304Z Queue Min Size: 4096(0x1000) 2022-11-23T01:33:53.7186659Z Queue Max Size: 131072(0x20000) 2022-11-23T01:33:53.7186993Z Queue Type: MULTI 2022-11-23T01:33:53.7187313Z Node: 7 2022-11-23T01:33:53.7187719Z Device Type: GPU 2022-11-23T01:33:53.7188041Z Cache Info: 2022-11-23T01:33:53.7188324Z L1: 16(0x10) KB 2022-11-23T01:33:53.7188641Z Chip ID: 26273(0x66a1) 2022-11-23T01:33:53.7188975Z Cacheline Size: 64(0x40) 2022-11-23T01:33:53.7189329Z Max Clock Freq. (MHz): 1725 2022-11-23T01:33:53.7189655Z BDFID: 26112 2022-11-23T01:33:53.7189975Z Internal Node ID: 7 2022-11-23T01:33:53.7190289Z Compute Unit: 60 2022-11-23T01:33:53.7190609Z SIMDs per CU: 4 2022-11-23T01:33:53.7190951Z Shader Engines: 4 2022-11-23T01:33:53.7191290Z Shader Arrs. per Eng.: 1 2022-11-23T01:33:53.7191638Z WatchPts on Addr. Ranges:4 2022-11-23T01:33:53.7191965Z Features: KERNEL_DISPATCH 2022-11-23T01:33:53.7192296Z Fast F16 Operation: FALSE 2022-11-23T01:33:53.7192630Z Wavefront Size: 64(0x40) 2022-11-23T01:33:53.7192975Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:33:53.7193301Z Workgroup Max Size per Dimension: 2022-11-23T01:33:53.7193646Z x 1024(0x400) 2022-11-23T01:33:53.7193954Z y 1024(0x400) 2022-11-23T01:33:53.7194240Z z 1024(0x400) 2022-11-23T01:33:53.7194579Z Max Waves Per CU: 40(0x28) 2022-11-23T01:33:53.7195025Z Max Work-item Per CU: 2560(0xa00) 2022-11-23T01:33:53.7195435Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:33:53.7195770Z Grid Max Size per Dimension: 2022-11-23T01:33:53.7196092Z x 4294967295(0xffffffff) 2022-11-23T01:33:53.7196417Z y 4294967295(0xffffffff) 2022-11-23T01:33:53.7196737Z z 4294967295(0xffffffff) 2022-11-23T01:33:53.7197144Z Max fbarriers/Workgrp: 32 2022-11-23T01:33:53.7197455Z Pool Info: 2022-11-23T01:33:53.7197734Z Pool 1 2022-11-23T01:33:53.7198088Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:33:53.7198452Z Size: 16760832(0xffc000) KB 2022-11-23T01:33:53.7198766Z Allocatable: TRUE 2022-11-23T01:33:53.7199104Z Alloc Granule: 4KB 2022-11-23T01:33:53.7199445Z Alloc Alignment: 4KB 2022-11-23T01:33:53.7199822Z Accessible by all: FALSE 2022-11-23T01:33:53.7200132Z Pool 2 2022-11-23T01:33:53.7200457Z Segment: GROUP 2022-11-23T01:33:53.7200766Z Size: 64(0x40) KB 2022-11-23T01:33:53.7201128Z Allocatable: FALSE 2022-11-23T01:33:53.7201486Z Alloc Granule: 0KB 2022-11-23T01:33:53.7201827Z Alloc Alignment: 0KB 2022-11-23T01:33:53.7202187Z Accessible by all: FALSE 2022-11-23T01:33:53.7202520Z ISA Info: 2022-11-23T01:33:53.7202777Z ISA 1 2022-11-23T01:33:53.7203229Z Name: amdgcn-amd-amdhsa--gfx906:sramecc+:xnack- 2022-11-23T01:33:53.7203686Z Machine Models: HSA_MACHINE_MODEL_LARGE 2022-11-23T01:33:53.7204068Z Profiles: HSA_PROFILE_BASE 2022-11-23T01:33:53.7204449Z Default Rounding Mode: NEAR 2022-11-23T01:33:53.7204804Z Default Rounding Mode: NEAR 2022-11-23T01:33:53.7205124Z Fast f16: TRUE 2022-11-23T01:33:53.7205460Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:33:53.7205816Z Workgroup Max Size per Dimension: 2022-11-23T01:33:53.7206143Z x 1024(0x400) 2022-11-23T01:33:53.7206464Z y 1024(0x400) 2022-11-23T01:33:53.7206789Z z 1024(0x400) 2022-11-23T01:33:53.7207244Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:33:53.7207553Z Grid Max Size per Dimension: 2022-11-23T01:33:53.7208249Z x 4294967295(0xffffffff) 2022-11-23T01:33:53.7208684Z y 4294967295(0xffffffff) 2022-11-23T01:33:53.7209094Z z 4294967295(0xffffffff) 2022-11-23T01:33:53.7209537Z FBarrier Max Size: 32 2022-11-23T01:33:53.7209909Z *** Done *** 2022-11-23T01:33:53.7268651Z ##[group]Run ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx') 2022-11-23T01:33:53.7269059Z ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx') 2022-11-23T01:33:53.7269400Z if [[ "x$ngpu" != "x2" && "x$ngpu" != "x4" ]]; then 2022-11-23T01:33:53.7269696Z  if [[ $ngpu -eq 0 ]]; then 2022-11-23T01:33:53.7270036Z  echo "Error: Failed to detect any GPUs on the runner" 2022-11-23T01:33:53.7270346Z  else 2022-11-23T01:33:53.7270676Z  echo "Error: Detected $ngpu GPUs on the runner, when only 2 or 4 were expected" 2022-11-23T01:33:53.7271023Z  fi 2022-11-23T01:33:53.7271459Z  echo "Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified" 2022-11-23T01:33:53.7271881Z  exit 1 2022-11-23T01:33:53.7272125Z fi 2022-11-23T01:33:53.7297015Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:33:53.7297363Z env: 2022-11-23T01:33:53.7297663Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:33:53.7298049Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:33:53.7298383Z ##[endgroup] 2022-11-23T01:33:53.8292180Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2022-11-23T01:33:53.8293231Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2022-11-23T01:33:53.8294175Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2022-11-23T01:33:53.8347779Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:33:53.8348527Z env: 2022-11-23T01:33:53.8349141Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:33:53.8349927Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:33:53.8350627Z ##[endgroup] 2022-11-23T01:33:53.8582365Z ##[group]Run # Examine the runner name. If it ends with "-2", this is the second runner on the host. 2022-11-23T01:33:53.8583275Z # Examine the runner name. If it ends with "-2", this is the second runner on the host. 2022-11-23T01:33:53.8583960Z if [[ worker-rocm-amd-94 == *-2 ]]; then 2022-11-23T01:33:53.8584538Z  # select the last two GPUs on the host 2022-11-23T01:33:53.8585473Z  echo "GPU_FLAG=--device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD130 --device=/dev/dri/renderD131 --group-add video --group-add daemon" >> "${GITHUB_ENV}" 2022-11-23T01:33:53.8586285Z else 2022-11-23T01:33:53.8586940Z  # select the first two GPUs on the host 2022-11-23T01:33:53.8587843Z  echo "GPU_FLAG=--device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon" >> "${GITHUB_ENV}" 2022-11-23T01:33:53.8588648Z fi 2022-11-23T01:33:53.8624403Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:33:53.8624738Z env: 2022-11-23T01:33:53.8625001Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:33:53.8625356Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:33:53.8625667Z ##[endgroup] 2022-11-23T01:33:53.8749946Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2022-11-23T01:33:53.8750850Z with: 2022-11-23T01:33:53.8752016Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm5.2-py3.8:072aae4a77ed7d3a69ad5683420509c41301b940 2022-11-23T01:33:53.8753181Z env: 2022-11-23T01:33:53.8753824Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:33:53.8754663Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:33:53.8755953Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T01:33:53.8757093Z ##[endgroup] 2022-11-23T01:33:53.8783661Z ##[group]Run retry () { "$@" || (sleep 1 && "$@") || (sleep 2 && "$@") } 2022-11-23T01:33:53.8784058Z retry () { "$@" || (sleep 1 && "$@") || (sleep 2 && "$@") } 2022-11-23T01:33:53.8784459Z # ignore output since only exit code is used for conditional 2022-11-23T01:33:53.8784905Z # only pull docker image if it's not available locally 2022-11-23T01:33:53.8785353Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2022-11-23T01:33:53.8785775Z  retry docker pull "${DOCKER_IMAGE}" 2022-11-23T01:33:53.8786070Z fi 2022-11-23T01:33:53.8806469Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:33:53.8806740Z env: 2022-11-23T01:33:53.8806968Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:33:53.8807260Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:33:53.8807786Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T01:33:53.8808412Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm5.2-py3.8:072aae4a77ed7d3a69ad5683420509c41301b940 2022-11-23T01:33:53.8808905Z ##[endgroup] 2022-11-23T01:33:53.9420133Z ##[group]Run python3 -m pip install psutil==5.9.1 2022-11-23T01:33:53.9421073Z python3 -m pip install psutil==5.9.1 2022-11-23T01:33:53.9421884Z python3 -m pip install pynvml==11.4.1 2022-11-23T01:33:53.9422770Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2022-11-23T01:33:53.9423733Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2022-11-23T01:33:53.9471531Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:33:53.9472043Z env: 2022-11-23T01:33:53.9472461Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:33:53.9473003Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:33:53.9473850Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T01:33:53.9474598Z ##[endgroup] 2022-11-23T01:33:54.8507192Z Collecting psutil==5.9.1 2022-11-23T01:33:55.1892947Z Installing collected packages: psutil 2022-11-23T01:33:55.3167036Z Successfully installed psutil-5.9.4 2022-11-23T01:33:56.2902897Z Collecting pynvml==11.4.1 2022-11-23T01:33:56.3569648Z Using cached https://files.pythonhosted.org/packages/cc/0a/47be6726fd13f1f4371fa858b506228ed12bc418c07ffcaa6c0f7ceedac0/pynvml-11.4.1-py3-none-any.whl 2022-11-23T01:33:56.3605252Z Installing collected packages: pynvml 2022-11-23T01:33:56.4180616Z Successfully installed pynvml-11.4.1 2022-11-23T01:33:56.4802104Z Prepare all required actions 2022-11-23T01:33:56.4803235Z Getting action download info 2022-11-23T01:33:56.7185980Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:4a8bfae15cc25cc0785c1603ee87a9da8fd442ea) 2022-11-23T01:33:57.7854981Z Download action repository 'actions/download-artifact@v3' (SHA:9782bd6a9848b53b110e712e20e42d89988822b7) 2022-11-23T01:33:58.5600196Z ##[group]Run ./.github/actions/download-build-artifacts 2022-11-23T01:33:58.5600524Z with: 2022-11-23T01:33:58.5600796Z name: linux-focal-rocm5.2-py3.8 2022-11-23T01:33:58.5601081Z env: 2022-11-23T01:33:58.5601345Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:33:58.5601672Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:33:58.5602174Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T01:33:58.5602619Z ##[endgroup] 2022-11-23T01:33:58.5637552Z ##[group]Run seemethere/download-artifact-s3@v4 2022-11-23T01:33:58.5637864Z with: 2022-11-23T01:33:58.5638153Z name: linux-focal-rocm5.2-py3.8 2022-11-23T01:33:58.5638449Z s3-bucket: gha-artifacts 2022-11-23T01:33:58.5638733Z region: us-east-1 2022-11-23T01:33:58.5638982Z env: 2022-11-23T01:33:58.5639239Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:33:58.5639567Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:33:58.5640056Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T01:33:58.5640509Z ##[endgroup] 2022-11-23T01:33:59.2528969Z Found 1 objects with prefix pytorch/pytorch/3528394938/linux-focal-rocm5.2-py3.8/ 2022-11-23T01:33:59.2530505Z Starting download (1/1): /home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip 2022-11-23T01:34:48.9214602Z Finished download (1/1): /home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip 2022-11-23T01:34:48.9215372Z 2022-11-23T01:34:48.9249708Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/ 2022-11-23T01:34:48.9270516Z Artifact download has finished successfully 2022-11-23T01:34:48.9360106Z ##[group]Run unzip -o artifacts.zip 2022-11-23T01:34:48.9360772Z unzip -o artifacts.zip 2022-11-23T01:34:48.9412249Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:34:48.9413029Z env: 2022-11-23T01:34:48.9413682Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:34:48.9414537Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:34:48.9415865Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T01:34:48.9417031Z ##[endgroup] 2022-11-23T01:34:48.9509192Z Archive: artifacts.zip 2022-11-23T01:34:48.9511376Z creating: dist/ 2022-11-23T01:34:50.2089693Z inflating: dist/torch-1.14.0a0+git1cfd385-cp38-cp38-linux_x86_64.whl 2022-11-23T01:34:50.2091727Z creating: build/custom_test_artifacts/ 2022-11-23T01:34:50.2093266Z creating: build/custom_test_artifacts/custom-op-build/ 2022-11-23T01:34:50.2094557Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2022-11-23T01:34:50.2096024Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeOutput.log 2022-11-23T01:34:50.2097470Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/ 2022-11-23T01:34:50.2098992Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeSystem.cmake 2022-11-23T01:34:50.2100645Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/ 2022-11-23T01:34:50.2102160Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/tmp/ 2022-11-23T01:34:50.2104435Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c 2022-11-23T01:34:50.2106279Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/a.out 2022-11-23T01:34:50.2107802Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/ 2022-11-23T01:34:50.2109357Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/ 2022-11-23T01:34:50.2111062Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp 2022-11-23T01:34:50.2112723Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out 2022-11-23T01:34:50.2114450Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin 2022-11-23T01:34:50.2116118Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake 2022-11-23T01:34:50.2117832Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin 2022-11-23T01:34:50.2119497Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake 2022-11-23T01:34:50.2120971Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2022-11-23T01:34:50.2122420Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeError.log 2022-11-23T01:34:50.2123910Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2022-11-23T01:34:50.2125365Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2022-11-23T01:34:50.2126954Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2022-11-23T01:34:50.2128806Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2022-11-23T01:34:50.2130417Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2022-11-23T01:34:50.2131981Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2022-11-23T01:34:50.2133588Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2022-11-23T01:34:50.2135174Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2022-11-23T01:34:50.2136781Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2022-11-23T01:34:50.2138375Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2022-11-23T01:34:50.2139971Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2022-11-23T01:34:50.2141575Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2022-11-23T01:34:50.2238330Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2022-11-23T01:34:50.2239593Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2022-11-23T01:34:50.2240790Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2022-11-23T01:34:50.2242040Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2022-11-23T01:34:50.2243221Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2022-11-23T01:34:50.2244381Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2022-11-23T01:34:50.2245602Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2022-11-23T01:34:50.2246780Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2022-11-23T01:34:50.2248383Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2022-11-23T01:34:50.2249657Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2022-11-23T01:34:50.2250874Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2022-11-23T01:34:50.2262603Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2022-11-23T01:34:50.2340337Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2022-11-23T01:34:50.2342145Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2022-11-23T01:34:50.2343788Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2022-11-23T01:34:50.2345341Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2022-11-23T01:34:50.2346839Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2022-11-23T01:34:50.2348329Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2022-11-23T01:34:50.2349768Z inflating: build/custom_test_artifacts/custom-op-build/detect_rocm_version.cc 2022-11-23T01:34:50.2351156Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2022-11-23T01:34:50.2352458Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2022-11-23T01:34:50.2353812Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2022-11-23T01:34:50.2434962Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2022-11-23T01:34:50.2492932Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2022-11-23T01:34:50.2494238Z creating: build/custom_test_artifacts/jit-hook-build/ 2022-11-23T01:34:50.2495458Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2022-11-23T01:34:50.2496900Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeOutput.log 2022-11-23T01:34:50.2498313Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/ 2022-11-23T01:34:50.2499775Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeSystem.cmake 2022-11-23T01:34:50.2501253Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/ 2022-11-23T01:34:50.2502728Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/tmp/ 2022-11-23T01:34:50.2504354Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c 2022-11-23T01:34:50.2505926Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/a.out 2022-11-23T01:34:50.2507385Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/ 2022-11-23T01:34:50.2508892Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/ 2022-11-23T01:34:50.2510566Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp 2022-11-23T01:34:50.2512229Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out 2022-11-23T01:34:50.2513860Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin 2022-11-23T01:34:50.2515466Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake 2022-11-23T01:34:50.2517126Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin 2022-11-23T01:34:50.2518774Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake 2022-11-23T01:34:50.2520182Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2022-11-23T01:34:50.2521586Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeError.log 2022-11-23T01:34:50.2523284Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2022-11-23T01:34:50.2524855Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2022-11-23T01:34:50.2526431Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2022-11-23T01:34:50.2528473Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2022-11-23T01:34:50.2530149Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2022-11-23T01:34:50.2531707Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2022-11-23T01:34:50.2533295Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2022-11-23T01:34:50.2534889Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2022-11-23T01:34:50.2536501Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2022-11-23T01:34:50.2537701Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2022-11-23T01:34:50.2538860Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2022-11-23T01:34:50.2540074Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2022-11-23T01:34:50.2595581Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2022-11-23T01:34:50.2598221Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2022-11-23T01:34:50.2599931Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2022-11-23T01:34:50.2601445Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2022-11-23T01:34:50.2602912Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2022-11-23T01:34:50.2604352Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2022-11-23T01:34:50.2605764Z inflating: build/custom_test_artifacts/jit-hook-build/detect_rocm_version.cc 2022-11-23T01:34:50.2607111Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2022-11-23T01:34:50.2608580Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2022-11-23T01:34:50.2609903Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2022-11-23T01:34:50.2647987Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2022-11-23T01:34:50.2649618Z creating: build/custom_test_artifacts/custom-backend-build/ 2022-11-23T01:34:50.2650947Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2022-11-23T01:34:50.2652447Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeOutput.log 2022-11-23T01:34:50.2653919Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/ 2022-11-23T01:34:50.2655474Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeSystem.cmake 2022-11-23T01:34:50.2657050Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/ 2022-11-23T01:34:50.2658609Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/tmp/ 2022-11-23T01:34:50.2660282Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c 2022-11-23T01:34:50.2661933Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/a.out 2022-11-23T01:34:50.2663481Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/ 2022-11-23T01:34:50.2665033Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/ 2022-11-23T01:34:50.2667106Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp 2022-11-23T01:34:50.2668993Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out 2022-11-23T01:34:50.2670698Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin 2022-11-23T01:34:50.2672396Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake 2022-11-23T01:34:50.2674140Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin 2022-11-23T01:34:50.2675841Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake 2022-11-23T01:34:50.2677356Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2022-11-23T01:34:50.2678856Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeError.log 2022-11-23T01:34:50.2680384Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2022-11-23T01:34:50.2681926Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2022-11-23T01:34:50.2683585Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2022-11-23T01:34:50.2685349Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2022-11-23T01:34:50.2687060Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2022-11-23T01:34:50.2688810Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2022-11-23T01:34:50.2690498Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2022-11-23T01:34:50.2692182Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2022-11-23T01:34:50.2693889Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2022-11-23T01:34:50.2695570Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2022-11-23T01:34:50.2697252Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2022-11-23T01:34:50.2699016Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2022-11-23T01:34:50.2811214Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2022-11-23T01:34:50.2813083Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2022-11-23T01:34:50.2814860Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2022-11-23T01:34:50.2816731Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2022-11-23T01:34:50.2818497Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2022-11-23T01:34:50.2820202Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2022-11-23T01:34:50.2821952Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2022-11-23T01:34:50.2823676Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2022-11-23T01:34:50.2825402Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2022-11-23T01:34:50.2827116Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2022-11-23T01:34:50.2828816Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2022-11-23T01:34:50.2834523Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2022-11-23T01:34:50.2889935Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2022-11-23T01:34:50.2892310Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2022-11-23T01:34:50.2894453Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2022-11-23T01:34:50.2896174Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2022-11-23T01:34:50.2897668Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2022-11-23T01:34:50.2899158Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2022-11-23T01:34:50.2900637Z inflating: build/custom_test_artifacts/custom-backend-build/detect_rocm_version.cc 2022-11-23T01:34:50.2902066Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2022-11-23T01:34:50.2903415Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2022-11-23T01:34:50.2904779Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2022-11-23T01:34:50.3001649Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2022-11-23T01:34:50.3041775Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2022-11-23T01:34:50.3042943Z creating: build/lib/ 2022-11-23T01:34:50.3043663Z inflating: build/lib/libclog.a 2022-11-23T01:34:50.3109920Z inflating: build/lib/libgtest.a 2022-11-23T01:34:50.3117421Z inflating: build/lib/libpthreadpool.a 2022-11-23T01:34:50.3195283Z inflating: build/lib/libbenchmark.a 2022-11-23T01:34:50.3291599Z inflating: build/lib/libprotobuf-lite.a 2022-11-23T01:34:50.3297125Z inflating: build/lib/libittnotify.a 2022-11-23T01:34:50.3328412Z inflating: build/lib/libtensorpipe_uv.a 2022-11-23T01:34:50.3395761Z inflating: build/lib/libasmjit.a 2022-11-23T01:34:50.3869448Z inflating: build/lib/libprotobuf.a 2022-11-23T01:34:50.3967203Z inflating: build/lib/libgloo.a 2022-11-23T01:34:50.3998618Z inflating: build/lib/libfmt.a 2022-11-23T01:34:50.3999654Z inflating: build/lib/libcaffe2_nvrtc.so 2022-11-23T01:34:50.4000473Z inflating: build/lib/libfoxi_loader.a 2022-11-23T01:34:50.4064305Z inflating: build/lib/libc10.so 2022-11-23T01:34:50.4065384Z inflating: build/lib/libtorch_global_deps.so 2022-11-23T01:34:50.4074757Z inflating: build/lib/libcpuinfo.a 2022-11-23T01:34:50.4075993Z inflating: build/lib/libnnpack_reference_layers.a 2022-11-23T01:34:50.4085885Z inflating: build/lib/libcpuinfo_internals.a 2022-11-23T01:34:50.4107129Z inflating: build/lib/libgmock.a 2022-11-23T01:34:50.4108208Z inflating: build/lib/libgtest_main.a 2022-11-23T01:34:50.4109031Z inflating: build/lib/libbenchmark_main.a 2022-11-23T01:34:51.2787224Z inflating: build/lib/libdnnl.a 2022-11-23T01:34:51.3309458Z inflating: build/lib/libprotoc.a 2022-11-23T01:34:51.3903591Z inflating: build/lib/libtensorpipe.a 2022-11-23T01:34:51.4309007Z inflating: build/lib/libgloo_hip.a 2022-11-23T01:34:51.4348088Z inflating: build/lib/libc10_hip.so 2022-11-23T01:34:51.4349157Z inflating: build/lib/libgmock_main.a 2022-11-23T01:34:51.5637038Z inflating: build/lib/libfbgemm.a 2022-11-23T01:34:51.5653145Z inflating: build/lib/libqnnpack.a 2022-11-23T01:34:51.6675192Z inflating: build/lib/libdnnl_graph.a 2022-11-23T01:34:51.6972984Z inflating: build/lib/libkineto.a 2022-11-23T01:34:51.6994305Z inflating: build/lib/libpytorch_qnnpack.a 2022-11-23T01:34:51.7036203Z inflating: build/lib/libcaffe2_protos.a 2022-11-23T01:34:51.7157018Z inflating: build/lib/libXNNPACK.a 2022-11-23T01:34:51.7204415Z inflating: build/lib/libonnx_proto.a 2022-11-23T01:34:51.7806032Z inflating: build/lib/libonnx.a 2022-11-23T01:34:51.7825454Z inflating: build/lib/libnnpack.a 2022-11-23T01:34:54.1964908Z inflating: build/lib/libtorch_cpu.so 2022-11-23T01:34:55.3730983Z inflating: build/lib/libtorch_hip.so 2022-11-23T01:34:55.3731838Z inflating: build/lib/libtorch.so 2022-11-23T01:34:55.3755117Z inflating: build/lib/libjitbackend_test.so 2022-11-23T01:34:55.3810050Z inflating: build/lib/libtorchbind_test.so 2022-11-23T01:34:55.3838098Z inflating: build/lib/libbackend_with_compiler.so 2022-11-23T01:34:55.3841701Z inflating: build/lib/libshm.so 2022-11-23T01:34:55.5544691Z inflating: build/lib/libtorch_python.so 2022-11-23T01:34:55.5582079Z inflating: build/lib/libnnapi_backend.so 2022-11-23T01:34:55.5583112Z creating: build/bin/ 2022-11-23T01:34:55.5583856Z creating: build/bin/CMakeFiles/ 2022-11-23T01:34:55.5584820Z inflating: build/bin/CMakeFiles/CMakeDirectoryInformation.cmake 2022-11-23T01:34:55.5585845Z extracting: build/bin/CMakeFiles/progress.marks 2022-11-23T01:34:55.5634801Z inflating: build/bin/hip_generator_test 2022-11-23T01:34:55.5635649Z inflating: build/bin/Makefile 2022-11-23T01:34:55.5688028Z inflating: build/bin/variant_test 2022-11-23T01:34:55.5688883Z inflating: build/bin/cmake_install.cmake 2022-11-23T01:34:55.5743881Z inflating: build/bin/undefined_tensor_test 2022-11-23T01:34:55.5744777Z inflating: build/bin/CTestTestfile.cmake 2022-11-23T01:34:55.5797112Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2022-11-23T01:34:55.5850789Z inflating: build/bin/c10_DeviceGuard_test 2022-11-23T01:34:55.5904630Z inflating: build/bin/c10_Device_test 2022-11-23T01:34:55.5966395Z inflating: build/bin/c10_DispatchKeySet_test 2022-11-23T01:34:55.6017767Z inflating: build/bin/c10_StreamGuard_test 2022-11-23T01:34:55.6070013Z inflating: build/bin/c10_SymInt_test 2022-11-23T01:34:55.6128486Z inflating: build/bin/c10_InlineDeviceGuard_test 2022-11-23T01:34:55.6186585Z inflating: build/bin/c10_InlineStreamGuard_test 2022-11-23T01:34:55.6245675Z inflating: build/bin/c10_SizesAndStrides_test 2022-11-23T01:34:55.6297186Z inflating: build/bin/c10_Array_test 2022-11-23T01:34:55.6352388Z inflating: build/bin/c10_Bitset_test 2022-11-23T01:34:55.6405249Z inflating: build/bin/c10_ConstexprCrc_test 2022-11-23T01:34:55.6459390Z inflating: build/bin/c10_C++17_test 2022-11-23T01:34:55.6511572Z inflating: build/bin/c10_DeadlockDetection_test 2022-11-23T01:34:55.6563133Z inflating: build/bin/c10_Half_test 2022-11-23T01:34:55.6623129Z inflating: build/bin/c10_LeftRight_test 2022-11-23T01:34:55.6691143Z inflating: build/bin/c10_Metaprogramming_test 2022-11-23T01:34:55.6743426Z inflating: build/bin/c10_Synchronized_test 2022-11-23T01:34:55.6902817Z inflating: build/bin/c10_SmallVectorTest 2022-11-23T01:34:55.6961855Z inflating: build/bin/c10_ThreadLocal_test 2022-11-23T01:34:55.7019842Z inflating: build/bin/c10_TypeIndex_test 2022-11-23T01:34:55.7068593Z inflating: build/bin/c10_TypeTraits_test 2022-11-23T01:34:55.7122095Z inflating: build/bin/c10_TypeList_test 2022-11-23T01:34:55.7176986Z inflating: build/bin/c10_accumulate_test 2022-11-23T01:34:55.7234716Z inflating: build/bin/c10_bfloat16_test 2022-11-23T01:34:55.7292612Z inflating: build/bin/c10_complex_math_test 2022-11-23T01:34:55.7345209Z inflating: build/bin/c10_flags_test 2022-11-23T01:34:55.7400914Z inflating: build/bin/c10_exception_test 2022-11-23T01:34:55.7458830Z inflating: build/bin/c10_complex_test 2022-11-23T01:34:55.7574123Z inflating: build/bin/c10_either_test 2022-11-23T01:34:55.7628200Z inflating: build/bin/c10_irange_test 2022-11-23T01:34:55.7686894Z inflating: build/bin/c10_logging_test 2022-11-23T01:34:55.7859848Z inflating: build/bin/c10_intrusive_ptr_test 2022-11-23T01:34:55.7937591Z inflating: build/bin/c10_optional_test 2022-11-23T01:34:55.7996582Z inflating: build/bin/c10_registry_test 2022-11-23T01:34:55.8063340Z inflating: build/bin/c10_ordered_preserving_dict_test 2022-11-23T01:34:55.8116597Z inflating: build/bin/c10_tempfile_test 2022-11-23T01:34:55.8179073Z inflating: build/bin/c10_string_view_test 2022-11-23T01:34:55.8238073Z inflating: build/bin/c10_typeid_test 2022-11-23T01:34:55.8290328Z inflating: build/bin/c10_intrusive_ptr_benchmark 2022-11-23T01:34:55.8755677Z inflating: build/bin/protoc-3.13.0.0 2022-11-23T01:34:55.9221245Z inflating: build/bin/protoc 2022-11-23T01:34:55.9273161Z inflating: build/bin/c10_hip_HIPTest 2022-11-23T01:34:55.9578901Z inflating: build/bin/vec_test_all_types_DEFAULT 2022-11-23T01:34:55.9905948Z inflating: build/bin/vec_test_all_types_AVX512 2022-11-23T01:34:56.0240464Z inflating: build/bin/vec_test_all_types_AVX2 2022-11-23T01:34:56.0296761Z inflating: build/bin/HashStoreTest 2022-11-23T01:34:56.0352019Z inflating: build/bin/FileStoreTest 2022-11-23T01:34:56.0415145Z inflating: build/bin/TCPStoreTest 2022-11-23T01:34:56.0417369Z inflating: build/bin/example_allreduce 2022-11-23T01:34:56.0486002Z inflating: build/bin/ProcessGroupGlooTest 2022-11-23T01:34:56.0540452Z inflating: build/bin/Dimname_test 2022-11-23T01:34:56.0617946Z inflating: build/bin/Dict_test 2022-11-23T01:34:56.0677663Z inflating: build/bin/NamedTensor_test 2022-11-23T01:34:56.0745223Z inflating: build/bin/MaybeOwned_test 2022-11-23T01:34:56.0808630Z inflating: build/bin/static_runtime_bench 2022-11-23T01:34:56.0869560Z inflating: build/bin/apply_utils_test 2022-11-23T01:34:56.0934210Z inflating: build/bin/basic 2022-11-23T01:34:56.0995761Z inflating: build/bin/atest 2022-11-23T01:34:56.1053095Z inflating: build/bin/broadcast_test 2022-11-23T01:34:56.1114714Z inflating: build/bin/cpu_generator_test 2022-11-23T01:34:56.1170534Z inflating: build/bin/cpu_profiling_allocator_test 2022-11-23T01:34:56.1439116Z inflating: build/bin/static_runtime_test 2022-11-23T01:34:56.1491405Z inflating: build/bin/dispatch_key_set_test 2022-11-23T01:34:56.1585545Z inflating: build/bin/cpu_rng_test 2022-11-23T01:34:56.1638544Z inflating: build/bin/dlconvertor_test 2022-11-23T01:34:56.1700422Z inflating: build/bin/extension_backend_test 2022-11-23T01:34:56.1756887Z inflating: build/bin/half_test 2022-11-23T01:34:56.1809335Z inflating: build/bin/lazy_tensor_test 2022-11-23T01:34:56.1865383Z inflating: build/bin/math_kernel_test 2022-11-23T01:34:56.1963736Z inflating: build/bin/ivalue_test 2022-11-23T01:34:56.2020683Z inflating: build/bin/memory_format_test 2022-11-23T01:34:56.2075695Z inflating: build/bin/memory_overlapping_test 2022-11-23T01:34:56.2131622Z inflating: build/bin/operator_name_test 2022-11-23T01:34:56.2184872Z inflating: build/bin/mobile_memory_cleanup 2022-11-23T01:34:56.2244154Z inflating: build/bin/native_test 2022-11-23T01:34:56.2299027Z inflating: build/bin/operators_test 2022-11-23T01:34:56.2353066Z inflating: build/bin/packedtensoraccessor_test 2022-11-23T01:34:56.2413558Z inflating: build/bin/quantized_test 2022-11-23T01:34:56.2466411Z inflating: build/bin/reduce_ops_test 2022-11-23T01:34:56.2520598Z inflating: build/bin/reportMemoryUsage_test 2022-11-23T01:34:56.2589052Z inflating: build/bin/pow_test 2022-11-23T01:34:56.2648640Z inflating: build/bin/scalar_tensor_test 2022-11-23T01:34:56.2708763Z inflating: build/bin/scalar_test 2022-11-23T01:34:56.2763751Z inflating: build/bin/stride_properties_test 2022-11-23T01:34:56.2765390Z inflating: build/bin/thread_init_test 2022-11-23T01:34:56.2826059Z inflating: build/bin/type_ptr_test 2022-11-23T01:34:56.2908306Z inflating: build/bin/tensor_iterator_test 2022-11-23T01:34:56.2965526Z inflating: build/bin/test_parallel 2022-11-23T01:34:56.3030290Z inflating: build/bin/type_test 2022-11-23T01:34:56.3031320Z inflating: build/bin/verify_api_visibility 2022-11-23T01:34:56.3105434Z inflating: build/bin/vmap_test 2022-11-23T01:34:56.3159996Z inflating: build/bin/weakref_test 2022-11-23T01:34:56.3213983Z inflating: build/bin/wrapdim_test 2022-11-23T01:34:56.3265339Z inflating: build/bin/xla_tensor_test 2022-11-23T01:34:56.3378161Z inflating: build/bin/List_test 2022-11-23T01:34:56.3441207Z inflating: build/bin/IListRef_test 2022-11-23T01:34:56.3566646Z inflating: build/bin/kernel_function_legacy_test 2022-11-23T01:34:56.3635758Z inflating: build/bin/KernelFunction_test 2022-11-23T01:34:56.3736509Z inflating: build/bin/kernel_function_test 2022-11-23T01:34:56.3869261Z inflating: build/bin/kernel_lambda_legacy_test 2022-11-23T01:34:56.3977139Z inflating: build/bin/kernel_lambda_test 2022-11-23T01:34:56.4040677Z inflating: build/bin/kernel_stackbased_test 2022-11-23T01:34:56.4140040Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2022-11-23T01:34:56.4195194Z inflating: build/bin/CppSignature_test 2022-11-23T01:34:56.4244911Z inflating: build/bin/op_allowlist_test 2022-11-23T01:34:56.4303075Z inflating: build/bin/backend_fallback_test 2022-11-23T01:34:56.4354215Z inflating: build/bin/hip_complex_math_test 2022-11-23T01:34:56.4409039Z inflating: build/bin/inline_container_test 2022-11-23T01:34:56.4717360Z inflating: build/bin/op_registration_test 2022-11-23T01:34:56.4769303Z inflating: build/bin/hip_complex_test 2022-11-23T01:34:56.4825847Z inflating: build/bin/hip_apply_test 2022-11-23T01:34:56.4876571Z inflating: build/bin/hip_distributions_test 2022-11-23T01:34:56.4927466Z inflating: build/bin/hip_half_test 2022-11-23T01:34:56.4978046Z inflating: build/bin/hip_integer_divider_test 2022-11-23T01:34:56.5030181Z inflating: build/bin/hip_optional_test 2022-11-23T01:34:56.5082007Z inflating: build/bin/hip_packedtensoraccessor_test 2022-11-23T01:34:56.5136486Z inflating: build/bin/hip_dlconvertor_test 2022-11-23T01:34:56.5186563Z inflating: build/bin/hip_vectorized_test 2022-11-23T01:34:56.5203596Z inflating: build/bin/tutorial_tensorexpr 2022-11-23T01:34:56.5261149Z inflating: build/bin/test_dist_autograd 2022-11-23T01:34:56.5334320Z inflating: build/bin/test_cpp_rpc 2022-11-23T01:34:56.6163033Z inflating: build/bin/test_tensorexpr 2022-11-23T01:34:56.6164511Z inflating: build/bin/parallel_benchmark 2022-11-23T01:34:56.6236407Z inflating: build/bin/test_mobile_nnc 2022-11-23T01:34:56.6247368Z inflating: build/bin/aot_model_compiler_test 2022-11-23T01:34:56.6610293Z inflating: build/bin/test_lazy 2022-11-23T01:34:56.7783596Z inflating: build/bin/test_api 2022-11-23T01:34:56.7787833Z inflating: build/bin/torch_shm_manager 2022-11-23T01:34:56.8370204Z inflating: build/bin/test_jit 2022-11-23T01:34:56.8371925Z inflating: .pytorch-test-times.json 2022-11-23T01:34:56.8416713Z ##[group]Run df -H 2022-11-23T01:34:56.8417361Z df -H 2022-11-23T01:34:56.8463174Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:34:56.8463790Z env: 2022-11-23T01:34:56.8464307Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:34:56.8464967Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:34:56.8465979Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T01:34:56.8466871Z ##[endgroup] 2022-11-23T01:34:56.8549741Z Filesystem Size Used Avail Use% Mounted on 2022-11-23T01:34:56.8551226Z udev 271G 0 271G 0% /dev 2022-11-23T01:34:56.8552176Z tmpfs 55G 1.6M 55G 1% /run 2022-11-23T01:34:56.8553518Z /dev/mapper/ubuntu--server--x8664--vg-root 1.9T 203G 1.6T 12% / 2022-11-23T01:34:56.8554888Z tmpfs 271G 17k 271G 1% /dev/shm 2022-11-23T01:34:56.8556586Z tmpfs 5.3M 0 5.3M 0% /run/lock 2022-11-23T01:34:56.8557941Z tmpfs 271G 0 271G 0% /sys/fs/cgroup 2022-11-23T01:34:56.8558970Z /dev/nvme0n1p1 755M 596M 121M 84% /boot 2022-11-23T01:34:56.8559806Z tmpfs 55G 13k 55G 1% /run/user/1121 2022-11-23T01:34:56.8560653Z tmpfs 55G 0 55G 0% /run/user/1000 2022-11-23T01:34:56.8608782Z ##[group]Run .github/scripts/parse_ref.py 2022-11-23T01:34:56.8609632Z .github/scripts/parse_ref.py 2022-11-23T01:34:56.8663569Z shell: /bin/bash -e {0} 2022-11-23T01:34:56.8663858Z env: 2022-11-23T01:34:56.8664130Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:34:56.8664424Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:34:56.8664891Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T01:34:56.8665314Z ##[endgroup] 2022-11-23T01:34:56.9061472Z ##[group]Run set -x 2022-11-23T01:34:56.9062295Z set -x 2022-11-23T01:34:56.9062874Z  2022-11-23T01:34:56.9063579Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2022-11-23T01:34:56.9064465Z  TEST_COMMAND=.jenkins/pytorch/multigpu-test.sh 2022-11-23T01:34:56.9065375Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2022-11-23T01:34:56.9066215Z  TEST_COMMAND=.jenkins/caffe2/test.sh 2022-11-23T01:34:56.9066906Z else 2022-11-23T01:34:56.9067620Z  TEST_COMMAND=.jenkins/pytorch/test.sh 2022-11-23T01:34:56.9068321Z fi 2022-11-23T01:34:56.9068856Z  2022-11-23T01:34:56.9069696Z COMMIT_MESSAGES=$(git cherry -v "origin/${GIT_DEFAULT_BRANCH:-master}") 2022-11-23T01:34:56.9070522Z  2022-11-23T01:34:56.9071286Z # sanitize the input commit message and PR body here: 2022-11-23T01:34:56.9072029Z # 2022-11-23T01:34:56.9072957Z # trim all new lines from commit messages + PR_BODY to avoid issues with batch environment 2022-11-23T01:34:56.9074227Z # variable copying. see https://github.com/pytorch/pytorch/pull/80043#issuecomment-1167796028 2022-11-23T01:34:56.9075321Z COMMIT_MESSAGES="${COMMIT_MESSAGES//[$'\n\r']}" 2022-11-23T01:34:56.9076134Z PR_BODY="${PR_BODY//[$'\n\r']}" 2022-11-23T01:34:56.9076792Z  2022-11-23T01:34:56.9077706Z # then trim all special characters like single and double quotes to avoid unescaped inputs to 2022-11-23T01:34:56.9078680Z # wreak havoc internally 2022-11-23T01:34:56.9079507Z export COMMIT_MESSAGES="${COMMIT_MESSAGES//[\'\"]}" 2022-11-23T01:34:56.9080354Z export PR_BODY="${PR_BODY//[\'\"]}" 2022-11-23T01:34:56.9081026Z  2022-11-23T01:34:56.9081821Z # detached container should get cleaned up by teardown_ec2_linux 2022-11-23T01:34:56.9082858Z # TODO: Stop building test binaries as part of the build phase 2022-11-23T01:34:56.9083824Z # Used for GPU_FLAG since that doesn't play nice 2022-11-23T01:34:56.9084669Z # shellcheck disable=SC2086,SC2090 2022-11-23T01:34:56.9085433Z container_name=$(docker run \ 2022-11-23T01:34:56.9086154Z  ${GPU_FLAG:-} \ 2022-11-23T01:34:56.9086841Z  -e BUILD_ENVIRONMENT \ 2022-11-23T01:34:56.9087537Z  -e PR_NUMBER \ 2022-11-23T01:34:56.9088374Z  -e GITHUB_ACTIONS \ 2022-11-23T01:34:56.9089038Z  -e BRANCH \ 2022-11-23T01:34:56.9089651Z  -e SHA1 \ 2022-11-23T01:34:56.9090307Z  -e AWS_DEFAULT_REGION \ 2022-11-23T01:34:56.9091014Z  -e IN_WHEEL_TEST \ 2022-11-23T01:34:56.9091688Z  -e SHARD_NUMBER \ 2022-11-23T01:34:56.9092352Z  -e TEST_CONFIG \ 2022-11-23T01:34:56.9093027Z  -e NUM_TEST_SHARDS \ 2022-11-23T01:34:56.9093687Z  -e PR_BODY \ 2022-11-23T01:34:56.9094357Z  -e COMMIT_MESSAGES \ 2022-11-23T01:34:56.9095360Z  -e PYTORCH_RETRY_TEST_CASES \ 2022-11-23T01:34:56.9096176Z  -e PYTORCH_OVERRIDE_FLAKY_SIGNAL \ 2022-11-23T01:34:56.9097001Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2022-11-23T01:34:56.9097726Z  -e SCCACHE_BUCKET \ 2022-11-23T01:34:56.9098480Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2022-11-23T01:34:56.9099307Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2022-11-23T01:34:56.9100162Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2022-11-23T01:34:56.9101058Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2022-11-23T01:34:56.9101898Z  --ulimit stack=10485760:83886080 \ 2022-11-23T01:34:56.9102698Z  --security-opt seccomp=unconfined \ 2022-11-23T01:34:56.9103467Z  --cap-add=SYS_PTRACE \ 2022-11-23T01:34:56.9104171Z  --shm-size="8g" \ 2022-11-23T01:34:56.9104800Z  --tty \ 2022-11-23T01:34:56.9105419Z  --detach \ 2022-11-23T01:34:56.9106281Z  --name="${container_name}" \ 2022-11-23T01:34:56.9106983Z  --user jenkins \ 2022-11-23T01:34:56.9107823Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2022-11-23T01:34:56.9108721Z  -w /var/lib/jenkins/workspace \ 2022-11-23T01:34:56.9109455Z  "${DOCKER_IMAGE}" 2022-11-23T01:34:56.9110084Z ) 2022-11-23T01:34:56.9110773Z # save container name for later step 2022-11-23T01:34:56.9111663Z echo "CONTAINER_NAME=${container_name}" >> "$GITHUB_ENV" 2022-11-23T01:34:56.9112867Z # jenkins user does not have write permission to mounted workspace; work-around by copying within container to jenkins home 2022-11-23T01:34:56.9114333Z docker exec -t "${container_name}" sh -c "cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && ${TEST_COMMAND}" 2022-11-23T01:34:56.9154028Z shell: /bin/bash -e {0} 2022-11-23T01:34:56.9154258Z env: 2022-11-23T01:34:56.9154506Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:34:56.9154803Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:34:56.9155271Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T01:34:56.9155736Z BUILD_ENVIRONMENT: linux-focal-rocm5.2-py3.8 2022-11-23T01:34:56.9156017Z PR_NUMBER: 2022-11-23T01:34:56.9156236Z BRANCH: master 2022-11-23T01:34:56.9156506Z SHA1: 1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T01:34:56.9156786Z PYTORCH_RETRY_TEST_CASES: 1 2022-11-23T01:34:56.9157063Z PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1 2022-11-23T01:34:56.9157334Z TEST_CONFIG: distributed 2022-11-23T01:34:56.9157576Z SHARD_NUMBER: 1 2022-11-23T01:34:56.9157801Z NUM_TEST_SHARDS: 2 2022-11-23T01:34:56.9158029Z PR_BODY: 2022-11-23T01:34:56.9158320Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-11-23T01:34:56.9158819Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm5.2-py3.8:072aae4a77ed7d3a69ad5683420509c41301b940 2022-11-23T01:34:56.9159351Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-11-23T01:34:56.9159707Z PYTORCH_JIT_ENABLE_NVFUSER: 1 2022-11-23T01:34:56.9159980Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 1 2022-11-23T01:34:56.9160274Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2022-11-23T01:34:56.9160535Z ##[endgroup] 2022-11-23T01:34:56.9207121Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2022-11-23T01:34:56.9208886Z + [[ linux-focal-rocm5.2-py3.8 == *onnx* ]] 2022-11-23T01:34:56.9209721Z + TEST_COMMAND=.jenkins/pytorch/test.sh 2022-11-23T01:34:56.9213483Z ++ git cherry -v origin/master 2022-11-23T01:34:56.9245823Z + COMMIT_MESSAGES= 2022-11-23T01:34:56.9246502Z + COMMIT_MESSAGES= 2022-11-23T01:34:56.9247110Z + PR_BODY= 2022-11-23T01:34:56.9247880Z + export COMMIT_MESSAGES= 2022-11-23T01:34:56.9248554Z + COMMIT_MESSAGES= 2022-11-23T01:34:56.9249188Z + export PR_BODY= 2022-11-23T01:34:56.9249787Z + PR_BODY= 2022-11-23T01:34:56.9265983Z +++ nproc --ignore=2 2022-11-23T01:34:56.9292149Z ++ docker run --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e PR_BODY -e COMMIT_MESSAGES -e PYTORCH_RETRY_TEST_CASES -e PYTORCH_OVERRIDE_FLAKY_SIGNAL -e MAX_JOBS=62 -e SCCACHE_BUCKET -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS --env-file=/tmp/github_env_3528394938 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --shm-size=8g --tty --detach --name= --user jenkins -v /home/pytorchci/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm5.2-py3.8:072aae4a77ed7d3a69ad5683420509c41301b940 2022-11-23T01:34:57.9746303Z + container_name=7ae77914f0c0c5de7f89cc247b24f443680151b55b56bc99ab51a9510965bce3 2022-11-23T01:34:57.9747511Z + echo CONTAINER_NAME=7ae77914f0c0c5de7f89cc247b24f443680151b55b56bc99ab51a9510965bce3 2022-11-23T01:34:57.9749932Z + docker exec -t 7ae77914f0c0c5de7f89cc247b24f443680151b55b56bc99ab51a9510965bce3 sh -c 'cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && .jenkins/pytorch/test.sh' 2022-11-23T01:35:03.8960187Z Processing ./dist/torch-1.14.0a0+git1cfd385-cp38-cp38-linux_x86_64.whl 2022-11-23T01:35:04.1256203Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.8/site-packages (from torch==1.14.0a0+git1cfd385) (4.4.0) 2022-11-23T01:35:04.1258100Z Requirement already satisfied: sympy in /opt/conda/lib/python3.8/site-packages (from torch==1.14.0a0+git1cfd385) (1.11.1) 2022-11-23T01:35:04.1260152Z Requirement already satisfied: networkx in /opt/conda/lib/python3.8/site-packages (from torch==1.14.0a0+git1cfd385) (2.6.3) 2022-11-23T01:35:04.1464131Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.8/site-packages (from sympy->torch==1.14.0a0+git1cfd385) (1.2.1) 2022-11-23T01:35:05.0384221Z Installing collected packages: torch 2022-11-23T01:35:12.0119742Z Successfully installed torch-1.14.0a0+git1cfd385 2022-11-23T01:35:12.0896127Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2022-11-23T01:35:12.1157859Z + TORCH_INSTALL_DIR=/opt/conda/lib/python3.8/site-packages/torch 2022-11-23T01:35:12.1159191Z + TORCH_BIN_DIR=/opt/conda/lib/python3.8/site-packages/torch/bin 2022-11-23T01:35:12.1160401Z + TORCH_LIB_DIR=/opt/conda/lib/python3.8/site-packages/torch/lib 2022-11-23T01:35:12.1161640Z + TORCH_TEST_DIR=/opt/conda/lib/python3.8/site-packages/torch/test 2022-11-23T01:35:12.1162443Z + BUILD_DIR=build 2022-11-23T01:35:12.1163125Z + BUILD_RENAMED_DIR=build_renamed 2022-11-23T01:35:12.1163838Z + BUILD_BIN_DIR=build/bin 2022-11-23T01:35:12.1164498Z + export VALGRIND=ON 2022-11-23T01:35:12.1165128Z + VALGRIND=ON 2022-11-23T01:35:12.1166096Z + [[ linux-focal-rocm5.2-py3.8 == *clang9* ]] 2022-11-23T01:35:12.1167133Z + [[ linux-focal-rocm5.2-py3.8 != *bazel* ]] 2022-11-23T01:35:12.1168350Z ++ realpath build/custom_test_artifacts 2022-11-23T01:35:12.1188593Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/pytorch/build/custom_test_artifacts 2022-11-23T01:35:12.1191043Z ++ dirname .jenkins/pytorch/test.sh 2022-11-23T01:35:12.1214209Z + source .jenkins/pytorch/common.sh 2022-11-23T01:35:12.1215069Z +++ dirname .jenkins/pytorch/common.sh 2022-11-23T01:35:12.1234249Z ++ source .jenkins/pytorch/common_utils.sh 2022-11-23T01:35:12.1235292Z +++ declare -f -t trap_add 2022-11-23T01:35:12.1255025Z ++ set -ex 2022-11-23T01:35:12.1256001Z ++ [[ linux-focal-rocm5.2-py3.8 == *rocm* ]] 2022-11-23T01:35:12.1256794Z ++ unset HIP_PLATFORM 2022-11-23T01:35:12.1257522Z ++ export PYTORCH_TEST_WITH_ROCM=1 2022-11-23T01:35:12.1258279Z ++ PYTORCH_TEST_WITH_ROCM=1 2022-11-23T01:35:12.1258991Z ++ export HSAKMT_DEBUG_LEVEL=4 2022-11-23T01:35:12.1260185Z ++ HSAKMT_DEBUG_LEVEL=4 2022-11-23T01:35:12.1260935Z ++ export HSA_FORCE_FINE_GRAIN_PCIE=1 2022-11-23T01:35:12.1261686Z ++ HSA_FORCE_FINE_GRAIN_PCIE=1 2022-11-23T01:35:12.1262396Z ++ BUILD_TEST_LIBTORCH=0 2022-11-23T01:35:12.1263274Z + echo 'Environment variables' 2022-11-23T01:35:12.1264017Z Environment variables 2022-11-23T01:35:12.1264613Z + env 2022-11-23T01:35:12.1277364Z INSTALLED_DB=yes 2022-11-23T01:35:12.1278740Z GITHUB_WORKSPACE=/home/pytorchci/actions-runner/_work/pytorch/pytorch 2022-11-23T01:35:12.1279930Z BUILD_ENVIRONMENT=linux-focal-rocm5.2-py3.8 2022-11-23T01:35:12.1280766Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1 2022-11-23T01:35:12.1281476Z HOSTNAME=7ae77914f0c0 2022-11-23T01:35:12.1282949Z GITHUB_PATH=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/add_path_fc43a25d-35a4-4c51-8dc6-86a14ee1fb53 2022-11-23T01:35:12.1284000Z GITHUB_ACTION=__self 2022-11-23T01:35:12.1284709Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 2022-11-23T01:35:12.1285409Z GITHUB_RUN_NUMBER=3445 2022-11-23T01:35:12.1286431Z TEST_CONFIG=distributed 2022-11-23T01:35:12.1287225Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2022-11-23T01:35:12.1288154Z GITHUB_REF_TYPE=branch 2022-11-23T01:35:12.1290526Z *** 2022-11-23T01:35:12.1291329Z GITHUB_ACTIONS=true 2022-11-23T01:35:12.1292062Z SHA1=1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T01:35:12.1292871Z GITHUB_SHA=1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T01:35:12.1293651Z GITHUB_REF=refs/heads/master 2022-11-23T01:35:12.1294327Z SHARD_NUMBER=1 2022-11-23T01:35:12.1294988Z GITHUB_REF_PROTECTED=true 2022-11-23T01:35:12.1295678Z HOME=/var/lib/jenkins 2022-11-23T01:35:12.1296434Z GITHUB_API_URL=https://api.github.com 2022-11-23T01:35:12.1297242Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2022-11-23T01:35:12.1298068Z LANG=C.UTF-8 2022-11-23T01:35:12.1298708Z PYTORCH_TEST_WITH_ROCM=1 2022-11-23T01:35:12.1299373Z NUM_TEST_SHARDS=2 2022-11-23T01:35:12.1300809Z GITHUB_STATE=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/save_state_fc43a25d-35a4-4c51-8dc6-86a14ee1fb53 2022-11-23T01:35:12.1301888Z MAGMA_HOME=/opt/rocm/magma 2022-11-23T01:35:12.1302571Z PYTORCH_RETRY_TEST_CASES=1 2022-11-23T01:35:12.1304035Z GITHUB_ENV=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_env_fc43a25d-35a4-4c51-8dc6-86a14ee1fb53 2022-11-23T01:35:12.1305041Z HSAKMT_DEBUG_LEVEL=4 2022-11-23T01:35:12.1306263Z GITHUB_EVENT_PATH=/home/pytorchci/actions-runner/_work/_temp/_github_workflow/event.json 2022-11-23T01:35:12.1307235Z GITHUB_EVENT_NAME=schedule 2022-11-23T01:35:12.1307909Z GITHUB_RUN_ID=3528394938 2022-11-23T01:35:12.1309426Z GITHUB_STEP_SUMMARY=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/step_summary_fc43a25d-35a4-4c51-8dc6-86a14ee1fb53 2022-11-23T01:35:12.1310549Z GITHUB_ACTOR=pytorchmergebot 2022-11-23T01:35:12.1311226Z PR_NUMBER= 2022-11-23T01:35:12.1311830Z GITHUB_RUN_ATTEMPT=1 2022-11-23T01:35:12.1312457Z VALGRIND=ON 2022-11-23T01:35:12.1313222Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2022-11-23T01:35:12.1313981Z TERM=xterm 2022-11-23T01:35:12.1314606Z INSTALLED_VISION=yes 2022-11-23T01:35:12.1315238Z BRANCH=master 2022-11-23T01:35:12.1316534Z GITHUB_ACTION_PATH=/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-rocm 2022-11-23T01:35:12.1317600Z GITHUB_SERVER_URL=https://github.com 2022-11-23T01:35:12.1318350Z PYTORCH_ROCM_ARCH=gfx906 2022-11-23T01:35:12.1318968Z SHLVL=1 2022-11-23T01:35:12.1319520Z MAX_JOBS=62 2022-11-23T01:35:12.1320123Z COMMIT_MESSAGES= 2022-11-23T01:35:12.1320753Z GITHUB_REF_NAME=master 2022-11-23T01:35:12.1321957Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2022-11-23T01:35:12.1322852Z GITHUB_JOB=test 2022-11-23T01:35:12.1323543Z GITHUB_REPOSITORY=pytorch/pytorch 2022-11-23T01:35:12.1324300Z LC_ALL=C.UTF-8 2022-11-23T01:35:12.1324943Z GITHUB_RETENTION_DAYS=90 2022-11-23T01:35:12.1325637Z GITHUB_ACTION_REPOSITORY= 2022-11-23T01:35:12.1326842Z PATH=/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2022-11-23T01:35:12.1328346Z GITHUB_BASE_REF= 2022-11-23T01:35:12.1328958Z CI=true 2022-11-23T01:35:12.1329588Z HSA_FORCE_FINE_GRAIN_PCIE=1 2022-11-23T01:35:12.1330326Z GITHUB_REPOSITORY_OWNER=pytorch 2022-11-23T01:35:12.1331037Z INSTALLED_PROTOBUF=yes 2022-11-23T01:35:12.1331699Z GITHUB_HEAD_REF= 2022-11-23T01:35:12.1332327Z GITHUB_ACTION_REF= 2022-11-23T01:35:12.1333351Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2022-11-23T01:35:12.1334200Z GITHUB_WORKFLOW=periodic 2022-11-23T01:35:12.1334917Z DEBIAN_FRONTEND=noninteractive 2022-11-23T01:35:12.1336413Z GITHUB_OUTPUT=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_output_fc43a25d-35a4-4c51-8dc6-86a14ee1fb53 2022-11-23T01:35:12.1337481Z OLDPWD=/var/lib/jenkins 2022-11-23T01:35:12.1338099Z PR_BODY= 2022-11-23T01:35:12.1338677Z _=/usr/bin/env 2022-11-23T01:35:12.1339455Z + echo 'Testing pytorch' 2022-11-23T01:35:12.1340101Z Testing pytorch 2022-11-23T01:35:12.1340819Z + export LANG=C.UTF-8 2022-11-23T01:35:12.1341723Z + LANG=C.UTF-8 2022-11-23T01:35:12.1342338Z + PR_NUMBER= 2022-11-23T01:35:12.1343003Z + [[ distributed == \d\e\f\a\u\l\t ]] 2022-11-23T01:35:12.1343789Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2022-11-23T01:35:12.1344787Z + [[ linux-focal-rocm5.2-py3.8 == *rocm* ]] 2022-11-23T01:35:12.1345583Z + export HIP_VISIBLE_DEVICES=0,1 2022-11-23T01:35:12.1346306Z + HIP_VISIBLE_DEVICES=0,1 2022-11-23T01:35:12.1346994Z + [[ distributed == \s\l\o\w ]] 2022-11-23T01:35:12.1348056Z + [[ linux-focal-rocm5.2-py3.8 == *slow-gradcheck* ]] 2022-11-23T01:35:12.1349133Z + [[ linux-focal-rocm5.2-py3.8 == *cuda* ]] 2022-11-23T01:35:12.1350147Z + [[ linux-focal-rocm5.2-py3.8 == *rocm* ]] 2022-11-23T01:35:12.1351006Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2022-11-23T01:35:12.1351830Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2022-11-23T01:35:12.1352579Z + [[ distributed == *crossref* ]] 2022-11-23T01:35:12.1353297Z + [[ distributed == *dynamo* ]] 2022-11-23T01:35:12.1354056Z + [[ distributed == *inductor* ]] 2022-11-23T01:35:12.1355013Z + [[ linux-focal-rocm5.2-py3.8 == *rocm* ]] 2022-11-23T01:35:12.1355718Z + rocminfo 2022-11-23T01:35:12.1514180Z ROCk module is loaded 2022-11-23T01:35:12.2125134Z ===================== 2022-11-23T01:35:12.2125852Z HSA System Attributes 2022-11-23T01:35:12.2126511Z ===================== 2022-11-23T01:35:12.2127143Z Runtime Version: 1.1 2022-11-23T01:35:12.2128074Z System Timestamp Freq.: 1000.000000MHz 2022-11-23T01:35:12.2129043Z Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count) 2022-11-23T01:35:12.2129999Z Machine Model: LARGE 2022-11-23T01:35:12.2130845Z System Endianness: LITTLE 2022-11-23T01:35:12.2131339Z 2022-11-23T01:35:12.2131583Z ========== 2022-11-23T01:35:12.2132200Z HSA Agents 2022-11-23T01:35:12.2132821Z ========== 2022-11-23T01:35:12.2133424Z ******* 2022-11-23T01:35:12.2134082Z Agent 1 2022-11-23T01:35:12.2134686Z ******* 2022-11-23T01:35:12.2135795Z Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:35:12.2136986Z Uuid: CPU-XX 2022-11-23T01:35:12.2138047Z Marketing Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:35:12.2138912Z Vendor Name: CPU 2022-11-23T01:35:12.2139730Z Feature: None specified 2022-11-23T01:35:12.2140548Z Profile: FULL_PROFILE 2022-11-23T01:35:12.2141377Z Float Round Mode: NEAR 2022-11-23T01:35:12.2142193Z Max Queue Number: 0(0x0) 2022-11-23T01:35:12.2142973Z Queue Min Size: 0(0x0) 2022-11-23T01:35:12.2143772Z Queue Max Size: 0(0x0) 2022-11-23T01:35:12.2144893Z Queue Type: MULTI 2022-11-23T01:35:12.2145646Z Node: 0 2022-11-23T01:35:12.2146423Z Device Type: CPU 2022-11-23T01:35:12.2147124Z Cache Info: 2022-11-23T01:35:12.2147833Z L1: 32768(0x8000) KB 2022-11-23T01:35:12.2148612Z Chip ID: 0(0x0) 2022-11-23T01:35:12.2149408Z Cacheline Size: 64(0x40) 2022-11-23T01:35:12.2150203Z Max Clock Freq. (MHz): 2200 2022-11-23T01:35:12.2150972Z BDFID: 0 2022-11-23T01:35:12.2151742Z Internal Node ID: 0 2022-11-23T01:35:12.2152535Z Compute Unit: 16 2022-11-23T01:35:12.2153442Z SIMDs per CU: 0 2022-11-23T01:35:12.2154238Z Shader Engines: 0 2022-11-23T01:35:12.2155048Z Shader Arrs. per Eng.: 0 2022-11-23T01:35:12.2155877Z WatchPts on Addr. Ranges:1 2022-11-23T01:35:12.2156607Z Features: None 2022-11-23T01:35:12.2157270Z Pool Info: 2022-11-23T01:35:12.2157916Z Pool 1 2022-11-23T01:35:12.2158708Z Segment: GLOBAL; FLAGS: FINE GRAINED 2022-11-23T01:35:12.2159550Z Size: 131954832(0x7dd7890) KB 2022-11-23T01:35:12.2160353Z Allocatable: TRUE 2022-11-23T01:35:12.2161162Z Alloc Granule: 4KB 2022-11-23T01:35:12.2161983Z Alloc Alignment: 4KB 2022-11-23T01:35:12.2162818Z Accessible by all: TRUE 2022-11-23T01:35:12.2163999Z Pool 2 2022-11-23T01:35:12.2164792Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2022-11-23T01:35:12.2165647Z Size: 131954832(0x7dd7890) KB 2022-11-23T01:35:12.2166457Z Allocatable: TRUE 2022-11-23T01:35:12.2167279Z Alloc Granule: 4KB 2022-11-23T01:35:12.2168190Z Alloc Alignment: 4KB 2022-11-23T01:35:12.2169007Z Accessible by all: TRUE 2022-11-23T01:35:12.2169737Z Pool 3 2022-11-23T01:35:12.2170524Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:35:12.2171370Z Size: 131954832(0x7dd7890) KB 2022-11-23T01:35:12.2172182Z Allocatable: TRUE 2022-11-23T01:35:12.2173003Z Alloc Granule: 4KB 2022-11-23T01:35:12.2173801Z Alloc Alignment: 4KB 2022-11-23T01:35:12.2174625Z Accessible by all: TRUE 2022-11-23T01:35:12.2175353Z ISA Info: 2022-11-23T01:35:12.2175977Z ******* 2022-11-23T01:35:12.2176589Z Agent 2 2022-11-23T01:35:12.2177200Z ******* 2022-11-23T01:35:12.2178201Z Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:35:12.2179240Z Uuid: CPU-XX 2022-11-23T01:35:12.2180304Z Marketing Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:35:12.2181157Z Vendor Name: CPU 2022-11-23T01:35:12.2181968Z Feature: None specified 2022-11-23T01:35:12.2183021Z Profile: FULL_PROFILE 2022-11-23T01:35:12.2183827Z Float Round Mode: NEAR 2022-11-23T01:35:12.2184638Z Max Queue Number: 0(0x0) 2022-11-23T01:35:12.2185429Z Queue Min Size: 0(0x0) 2022-11-23T01:35:12.2186230Z Queue Max Size: 0(0x0) 2022-11-23T01:35:12.2187028Z Queue Type: MULTI 2022-11-23T01:35:12.2187787Z Node: 1 2022-11-23T01:35:12.2188529Z Device Type: CPU 2022-11-23T01:35:12.2189236Z Cache Info: 2022-11-23T01:35:12.2189963Z L1: 32768(0x8000) KB 2022-11-23T01:35:12.2190722Z Chip ID: 0(0x0) 2022-11-23T01:35:12.2191495Z Cacheline Size: 64(0x40) 2022-11-23T01:35:12.2192431Z Max Clock Freq. (MHz): 2200 2022-11-23T01:35:12.2193217Z BDFID: 0 2022-11-23T01:35:12.2193971Z Internal Node ID: 1 2022-11-23T01:35:12.2194749Z Compute Unit: 16 2022-11-23T01:35:12.2195518Z SIMDs per CU: 0 2022-11-23T01:35:12.2196309Z Shader Engines: 0 2022-11-23T01:35:12.2197107Z Shader Arrs. per Eng.: 0 2022-11-23T01:35:12.2197935Z WatchPts on Addr. Ranges:1 2022-11-23T01:35:12.2198664Z Features: None 2022-11-23T01:35:12.2199320Z Pool Info: 2022-11-23T01:35:12.2199973Z Pool 1 2022-11-23T01:35:12.2200755Z Segment: GLOBAL; FLAGS: FINE GRAINED 2022-11-23T01:35:12.2201609Z Size: 132087932(0x7df807c) KB 2022-11-23T01:35:12.2202419Z Allocatable: TRUE 2022-11-23T01:35:12.2203216Z Alloc Granule: 4KB 2022-11-23T01:35:12.2204027Z Alloc Alignment: 4KB 2022-11-23T01:35:12.2204856Z Accessible by all: TRUE 2022-11-23T01:35:12.2205659Z Pool 2 2022-11-23T01:35:12.2206476Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2022-11-23T01:35:12.2207332Z Size: 132087932(0x7df807c) KB 2022-11-23T01:35:12.2208450Z Allocatable: TRUE 2022-11-23T01:35:12.2209268Z Alloc Granule: 4KB 2022-11-23T01:35:12.2210117Z Alloc Alignment: 4KB 2022-11-23T01:35:12.2210963Z Accessible by all: TRUE 2022-11-23T01:35:12.2211715Z Pool 3 2022-11-23T01:35:12.2212539Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:35:12.2213373Z Size: 132087932(0x7df807c) KB 2022-11-23T01:35:12.2214219Z Allocatable: TRUE 2022-11-23T01:35:12.2215065Z Alloc Granule: 4KB 2022-11-23T01:35:12.2215885Z Alloc Alignment: 4KB 2022-11-23T01:35:12.2216746Z Accessible by all: TRUE 2022-11-23T01:35:12.2217471Z ISA Info: 2022-11-23T01:35:12.2218103Z ******* 2022-11-23T01:35:12.2218733Z Agent 3 2022-11-23T01:35:12.2219355Z ******* 2022-11-23T01:35:12.2220394Z Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:35:12.2221675Z Uuid: CPU-XX 2022-11-23T01:35:12.2222757Z Marketing Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:35:12.2223619Z Vendor Name: CPU 2022-11-23T01:35:12.2224428Z Feature: None specified 2022-11-23T01:35:12.2224863Z Profile: FULL_PROFILE 2022-11-23T01:35:12.2225246Z Float Round Mode: NEAR 2022-11-23T01:35:12.2225554Z Max Queue Number: 0(0x0) 2022-11-23T01:35:12.2225852Z Queue Min Size: 0(0x0) 2022-11-23T01:35:12.2226149Z Queue Max Size: 0(0x0) 2022-11-23T01:35:12.2226439Z Queue Type: MULTI 2022-11-23T01:35:12.2226725Z Node: 2 2022-11-23T01:35:12.2227071Z Device Type: CPU 2022-11-23T01:35:12.2227340Z Cache Info: 2022-11-23T01:35:12.2227617Z L1: 32768(0x8000) KB 2022-11-23T01:35:12.2227906Z Chip ID: 0(0x0) 2022-11-23T01:35:12.2228185Z Cacheline Size: 64(0x40) 2022-11-23T01:35:12.2228485Z Max Clock Freq. (MHz): 2200 2022-11-23T01:35:12.2228775Z BDFID: 0 2022-11-23T01:35:12.2229065Z Internal Node ID: 2 2022-11-23T01:35:12.2229364Z Compute Unit: 16 2022-11-23T01:35:12.2229653Z SIMDs per CU: 0 2022-11-23T01:35:12.2229941Z Shader Engines: 0 2022-11-23T01:35:12.2230252Z Shader Arrs. per Eng.: 0 2022-11-23T01:35:12.2230606Z WatchPts on Addr. Ranges:1 2022-11-23T01:35:12.2230995Z Features: None 2022-11-23T01:35:12.2231274Z Pool Info: 2022-11-23T01:35:12.2231541Z Pool 1 2022-11-23T01:35:12.2231861Z Segment: GLOBAL; FLAGS: FINE GRAINED 2022-11-23T01:35:12.2232391Z Size: 132112788(0x7dfe194) KB 2022-11-23T01:35:12.2232783Z Allocatable: TRUE 2022-11-23T01:35:12.2233147Z Alloc Granule: 4KB 2022-11-23T01:35:12.2233493Z Alloc Alignment: 4KB 2022-11-23T01:35:12.2233828Z Accessible by all: TRUE 2022-11-23T01:35:12.2234177Z Pool 2 2022-11-23T01:35:12.2234547Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2022-11-23T01:35:12.2234898Z Size: 132112788(0x7dfe194) KB 2022-11-23T01:35:12.2235231Z Allocatable: TRUE 2022-11-23T01:35:12.2235588Z Alloc Granule: 4KB 2022-11-23T01:35:12.2235903Z Alloc Alignment: 4KB 2022-11-23T01:35:12.2236257Z Accessible by all: TRUE 2022-11-23T01:35:12.2236573Z Pool 3 2022-11-23T01:35:12.2236902Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:35:12.2237281Z Size: 132112788(0x7dfe194) KB 2022-11-23T01:35:12.2237609Z Allocatable: TRUE 2022-11-23T01:35:12.2237926Z Alloc Granule: 4KB 2022-11-23T01:35:12.2238267Z Alloc Alignment: 4KB 2022-11-23T01:35:12.2238826Z Accessible by all: TRUE 2022-11-23T01:35:12.2239129Z ISA Info: 2022-11-23T01:35:12.2239393Z ******* 2022-11-23T01:35:12.2239650Z Agent 4 2022-11-23T01:35:12.2239887Z ******* 2022-11-23T01:35:12.2240331Z Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:35:12.2240750Z Uuid: CPU-XX 2022-11-23T01:35:12.2241177Z Marketing Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:35:12.2241537Z Vendor Name: CPU 2022-11-23T01:35:12.2241890Z Feature: None specified 2022-11-23T01:35:12.2242205Z Profile: FULL_PROFILE 2022-11-23T01:35:12.2242550Z Float Round Mode: NEAR 2022-11-23T01:35:12.2242965Z Max Queue Number: 0(0x0) 2022-11-23T01:35:12.2243325Z Queue Min Size: 0(0x0) 2022-11-23T01:35:12.2243671Z Queue Max Size: 0(0x0) 2022-11-23T01:35:12.2244011Z Queue Type: MULTI 2022-11-23T01:35:12.2244423Z Node: 3 2022-11-23T01:35:12.2244718Z Device Type: CPU 2022-11-23T01:35:12.2245045Z Cache Info: 2022-11-23T01:35:12.2245348Z L1: 32768(0x8000) KB 2022-11-23T01:35:12.2245688Z Chip ID: 0(0x0) 2022-11-23T01:35:12.2246024Z Cacheline Size: 64(0x40) 2022-11-23T01:35:12.2246371Z Max Clock Freq. (MHz): 2200 2022-11-23T01:35:12.2246667Z BDFID: 0 2022-11-23T01:35:12.2246996Z Internal Node ID: 3 2022-11-23T01:35:12.2247330Z Compute Unit: 16 2022-11-23T01:35:12.2247668Z SIMDs per CU: 0 2022-11-23T01:35:12.2248042Z Shader Engines: 0 2022-11-23T01:35:12.2248393Z Shader Arrs. per Eng.: 0 2022-11-23T01:35:12.2248718Z WatchPts on Addr. Ranges:1 2022-11-23T01:35:12.2249027Z Features: None 2022-11-23T01:35:12.2249338Z Pool Info: 2022-11-23T01:35:12.2249619Z Pool 1 2022-11-23T01:35:12.2250069Z Segment: GLOBAL; FLAGS: FINE GRAINED 2022-11-23T01:35:12.2250431Z Size: 132111260(0x7dfdb9c) KB 2022-11-23T01:35:12.2250770Z Allocatable: TRUE 2022-11-23T01:35:12.2251122Z Alloc Granule: 4KB 2022-11-23T01:35:12.2251474Z Alloc Alignment: 4KB 2022-11-23T01:35:12.2251813Z Accessible by all: TRUE 2022-11-23T01:35:12.2252136Z Pool 2 2022-11-23T01:35:12.2252461Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2022-11-23T01:35:12.2252799Z Size: 132111260(0x7dfdb9c) KB 2022-11-23T01:35:12.2253133Z Allocatable: TRUE 2022-11-23T01:35:12.2253468Z Alloc Granule: 4KB 2022-11-23T01:35:12.2253819Z Alloc Alignment: 4KB 2022-11-23T01:35:12.2254174Z Accessible by all: TRUE 2022-11-23T01:35:12.2254481Z Pool 3 2022-11-23T01:35:12.2254894Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:35:12.2255244Z Size: 132111260(0x7dfdb9c) KB 2022-11-23T01:35:12.2255596Z Allocatable: TRUE 2022-11-23T01:35:12.2256058Z Alloc Granule: 4KB 2022-11-23T01:35:12.2256394Z Alloc Alignment: 4KB 2022-11-23T01:35:12.2256751Z Accessible by all: TRUE 2022-11-23T01:35:12.2257062Z ISA Info: 2022-11-23T01:35:12.2257303Z ******* 2022-11-23T01:35:12.2257563Z Agent 5 2022-11-23T01:35:12.2257819Z ******* 2022-11-23T01:35:12.2258140Z Name: gfx906 2022-11-23T01:35:12.2258569Z Uuid: GPU-621e518172da5ee8 2022-11-23T01:35:12.2258901Z Marketing Name: 2022-11-23T01:35:12.2259270Z Vendor Name: AMD 2022-11-23T01:35:12.2259645Z Feature: KERNEL_DISPATCH 2022-11-23T01:35:12.2259989Z Profile: BASE_PROFILE 2022-11-23T01:35:12.2260329Z Float Round Mode: NEAR 2022-11-23T01:35:12.2260663Z Max Queue Number: 128(0x80) 2022-11-23T01:35:12.2261021Z Queue Min Size: 64(0x40) 2022-11-23T01:35:12.2261327Z Queue Max Size: 131072(0x20000) 2022-11-23T01:35:12.2291389Z Queue Type: MULTI 2022-11-23T01:35:12.2291793Z Node: 4 2022-11-23T01:35:12.2292229Z Device Type: GPU 2022-11-23T01:35:12.2292618Z Cache Info: 2022-11-23T01:35:12.2293050Z L1: 16(0x10) KB 2022-11-23T01:35:12.2293508Z Chip ID: 26273(0x66a1) 2022-11-23T01:35:12.2294003Z Cacheline Size: 64(0x40) 2022-11-23T01:35:12.2294483Z Max Clock Freq. (MHz): 1725 2022-11-23T01:35:12.2295037Z BDFID: 8960 2022-11-23T01:35:12.2295624Z Internal Node ID: 4 2022-11-23T01:35:12.2296061Z Compute Unit: 60 2022-11-23T01:35:12.2296364Z SIMDs per CU: 4 2022-11-23T01:35:12.2296671Z Shader Engines: 4 2022-11-23T01:35:12.2296981Z Shader Arrs. per Eng.: 1 2022-11-23T01:35:12.2297292Z WatchPts on Addr. Ranges:4 2022-11-23T01:35:12.2297610Z Features: KERNEL_DISPATCH 2022-11-23T01:35:12.2297911Z Fast F16 Operation: TRUE 2022-11-23T01:35:12.2298210Z Wavefront Size: 64(0x40) 2022-11-23T01:35:12.2298521Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:35:12.2298817Z Workgroup Max Size per Dimension: 2022-11-23T01:35:12.2299115Z x 1024(0x400) 2022-11-23T01:35:12.2299381Z y 1024(0x400) 2022-11-23T01:35:12.2299655Z z 1024(0x400) 2022-11-23T01:35:12.2299959Z Max Waves Per CU: 40(0x28) 2022-11-23T01:35:12.2300482Z Max Work-item Per CU: 2560(0xa00) 2022-11-23T01:35:12.2300795Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:35:12.2301093Z Grid Max Size per Dimension: 2022-11-23T01:35:12.2301638Z x 4294967295(0xffffffff) 2022-11-23T01:35:12.2301931Z y 4294967295(0xffffffff) 2022-11-23T01:35:12.2302232Z z 4294967295(0xffffffff) 2022-11-23T01:35:12.2302540Z Max fbarriers/Workgrp: 32 2022-11-23T01:35:12.2302809Z Pool Info: 2022-11-23T01:35:12.2303048Z Pool 1 2022-11-23T01:35:12.2303334Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:35:12.2303649Z Size: 16760832(0xffc000) KB 2022-11-23T01:35:12.2303961Z Allocatable: TRUE 2022-11-23T01:35:12.2304267Z Alloc Granule: 4KB 2022-11-23T01:35:12.2304570Z Alloc Alignment: 4KB 2022-11-23T01:35:12.2304889Z Accessible by all: FALSE 2022-11-23T01:35:12.2305225Z Pool 2 2022-11-23T01:35:12.2305526Z Segment: GLOBAL; FLAGS: FINE GRAINED 2022-11-23T01:35:12.2305843Z Size: 16760832(0xffc000) KB 2022-11-23T01:35:12.2306149Z Allocatable: TRUE 2022-11-23T01:35:12.2306454Z Alloc Granule: 4KB 2022-11-23T01:35:12.2306758Z Alloc Alignment: 4KB 2022-11-23T01:35:12.2307065Z Accessible by all: FALSE 2022-11-23T01:35:12.2307357Z Pool 3 2022-11-23T01:35:12.2307633Z Segment: GROUP 2022-11-23T01:35:12.2307923Z Size: 64(0x40) KB 2022-11-23T01:35:12.2308221Z Allocatable: FALSE 2022-11-23T01:35:12.2308533Z Alloc Granule: 0KB 2022-11-23T01:35:12.2308857Z Alloc Alignment: 0KB 2022-11-23T01:35:12.2309158Z Accessible by all: FALSE 2022-11-23T01:35:12.2309438Z ISA Info: 2022-11-23T01:35:12.2309721Z ISA 1 2022-11-23T01:35:12.2310209Z Name: amdgcn-amd-amdhsa--gfx906:sramecc+:xnack- 2022-11-23T01:35:12.2310631Z Machine Models: HSA_MACHINE_MODEL_LARGE 2022-11-23T01:35:12.2311027Z Profiles: HSA_PROFILE_BASE 2022-11-23T01:35:12.2311404Z Default Rounding Mode: NEAR 2022-11-23T01:35:12.2311783Z Default Rounding Mode: NEAR 2022-11-23T01:35:12.2312153Z Fast f16: TRUE 2022-11-23T01:35:12.2312517Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:35:12.2312872Z Workgroup Max Size per Dimension: 2022-11-23T01:35:12.2313219Z x 1024(0x400) 2022-11-23T01:35:12.2313549Z y 1024(0x400) 2022-11-23T01:35:12.2313886Z z 1024(0x400) 2022-11-23T01:35:12.2314246Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:35:12.2314591Z Grid Max Size per Dimension: 2022-11-23T01:35:12.2314932Z x 4294967295(0xffffffff) 2022-11-23T01:35:12.2315285Z y 4294967295(0xffffffff) 2022-11-23T01:35:12.2315645Z z 4294967295(0xffffffff) 2022-11-23T01:35:12.2315997Z FBarrier Max Size: 32 2022-11-23T01:35:12.2316307Z ******* 2022-11-23T01:35:12.2316659Z Agent 6 2022-11-23T01:35:12.2316930Z ******* 2022-11-23T01:35:12.2317238Z Name: gfx906 2022-11-23T01:35:12.2317697Z Uuid: GPU-4410584172da5ebc 2022-11-23T01:35:12.2318052Z Marketing Name: 2022-11-23T01:35:12.2318403Z Vendor Name: AMD 2022-11-23T01:35:12.2318767Z Feature: KERNEL_DISPATCH 2022-11-23T01:35:12.2319149Z Profile: BASE_PROFILE 2022-11-23T01:35:12.2319511Z Float Round Mode: NEAR 2022-11-23T01:35:12.2319870Z Max Queue Number: 128(0x80) 2022-11-23T01:35:12.2320230Z Queue Min Size: 64(0x40) 2022-11-23T01:35:12.2320593Z Queue Max Size: 131072(0x20000) 2022-11-23T01:35:12.2321014Z Queue Type: MULTI 2022-11-23T01:35:12.2321357Z Node: 5 2022-11-23T01:35:12.2321706Z Device Type: GPU 2022-11-23T01:35:12.2322024Z Cache Info: 2022-11-23T01:35:12.2322337Z L1: 16(0x10) KB 2022-11-23T01:35:12.2322677Z Chip ID: 26273(0x66a1) 2022-11-23T01:35:12.2323025Z Cacheline Size: 64(0x40) 2022-11-23T01:35:12.2323378Z Max Clock Freq. (MHz): 1725 2022-11-23T01:35:12.2323725Z BDFID: 9728 2022-11-23T01:35:12.2324071Z Internal Node ID: 5 2022-11-23T01:35:12.2324417Z Compute Unit: 60 2022-11-23T01:35:12.2324749Z SIMDs per CU: 4 2022-11-23T01:35:12.2325051Z Shader Engines: 4 2022-11-23T01:35:12.2325358Z Shader Arrs. per Eng.: 1 2022-11-23T01:35:12.2325664Z WatchPts on Addr. Ranges:4 2022-11-23T01:35:12.2325957Z Features: KERNEL_DISPATCH 2022-11-23T01:35:12.2326247Z Fast F16 Operation: TRUE 2022-11-23T01:35:12.2326544Z Wavefront Size: 64(0x40) 2022-11-23T01:35:12.2326841Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:35:12.2327134Z Workgroup Max Size per Dimension: 2022-11-23T01:35:12.2327422Z x 1024(0x400) 2022-11-23T01:35:12.2327885Z y 1024(0x400) 2022-11-23T01:35:12.2328400Z z 1024(0x400) 2022-11-23T01:35:12.2328692Z Max Waves Per CU: 40(0x28) 2022-11-23T01:35:12.2329100Z Max Work-item Per CU: 2560(0xa00) 2022-11-23T01:35:12.2329413Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:35:12.2329696Z Grid Max Size per Dimension: 2022-11-23T01:35:12.2329980Z x 4294967295(0xffffffff) 2022-11-23T01:35:12.2330270Z y 4294967295(0xffffffff) 2022-11-23T01:35:12.2330554Z z 4294967295(0xffffffff) 2022-11-23T01:35:12.2330859Z Max fbarriers/Workgrp: 32 2022-11-23T01:35:12.2331132Z Pool Info: 2022-11-23T01:35:12.2331376Z Pool 1 2022-11-23T01:35:12.2331672Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:35:12.2331995Z Size: 16760832(0xffc000) KB 2022-11-23T01:35:12.2332386Z Allocatable: TRUE 2022-11-23T01:35:12.2332695Z Alloc Granule: 4KB 2022-11-23T01:35:12.2333004Z Alloc Alignment: 4KB 2022-11-23T01:35:12.2333313Z Accessible by all: FALSE 2022-11-23T01:35:12.2333584Z Pool 2 2022-11-23T01:35:12.2333881Z Segment: GLOBAL; FLAGS: FINE GRAINED 2022-11-23T01:35:12.2334186Z Size: 16760832(0xffc000) KB 2022-11-23T01:35:12.2334482Z Allocatable: TRUE 2022-11-23T01:35:12.2334788Z Alloc Granule: 4KB 2022-11-23T01:35:12.2335088Z Alloc Alignment: 4KB 2022-11-23T01:35:12.2335396Z Accessible by all: FALSE 2022-11-23T01:35:12.2335673Z Pool 3 2022-11-23T01:35:12.2336038Z Segment: GROUP 2022-11-23T01:35:12.2336326Z Size: 64(0x40) KB 2022-11-23T01:35:12.2336622Z Allocatable: FALSE 2022-11-23T01:35:12.2336934Z Alloc Granule: 0KB 2022-11-23T01:35:12.2337234Z Alloc Alignment: 0KB 2022-11-23T01:35:12.2337550Z Accessible by all: FALSE 2022-11-23T01:35:12.2337825Z ISA Info: 2022-11-23T01:35:12.2338058Z ISA 1 2022-11-23T01:35:12.2338468Z Name: amdgcn-amd-amdhsa--gfx906:sramecc+:xnack- 2022-11-23T01:35:12.2338826Z Machine Models: HSA_MACHINE_MODEL_LARGE 2022-11-23T01:35:12.2339158Z Profiles: HSA_PROFILE_BASE 2022-11-23T01:35:12.2339484Z Default Rounding Mode: NEAR 2022-11-23T01:35:12.2339805Z Default Rounding Mode: NEAR 2022-11-23T01:35:12.2340099Z Fast f16: TRUE 2022-11-23T01:35:12.2340398Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:35:12.2340692Z Workgroup Max Size per Dimension: 2022-11-23T01:35:12.2340986Z x 1024(0x400) 2022-11-23T01:35:12.2341263Z y 1024(0x400) 2022-11-23T01:35:12.2341541Z z 1024(0x400) 2022-11-23T01:35:12.2341828Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:35:12.2342116Z Grid Max Size per Dimension: 2022-11-23T01:35:12.2342406Z x 4294967295(0xffffffff) 2022-11-23T01:35:12.2342708Z y 4294967295(0xffffffff) 2022-11-23T01:35:12.2343003Z z 4294967295(0xffffffff) 2022-11-23T01:35:12.2343344Z FBarrier Max Size: 32 2022-11-23T01:35:12.2343609Z *** Done *** 2022-11-23T01:35:12.2343864Z + rocminfo 2022-11-23T01:35:12.2344174Z + grep -E 'Name:.*\sgfx|Marketing' 2022-11-23T01:35:12.2899706Z Marketing Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:35:12.2901028Z Marketing Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:35:12.2902242Z Marketing Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:35:12.2903443Z Marketing Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:35:12.2904320Z Name: gfx906 2022-11-23T01:35:12.2905117Z Marketing Name: 2022-11-23T01:35:12.2905965Z Name: gfx906 2022-11-23T01:35:12.2907066Z Marketing Name: 2022-11-23T01:35:12.3070938Z + [[ linux-focal-rocm5.2-py3.8 != *-bazel-* ]] 2022-11-23T01:35:12.3072041Z + pip_install --user ninja==1.10.2 2022-11-23T01:35:12.3073165Z + pip install --progress-bar off --user ninja==1.10.2 2022-11-23T01:35:12.8735597Z Collecting ninja==1.10.2 2022-11-23T01:35:12.9220434Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2022-11-23T01:35:13.7779037Z Installing collected packages: ninja 2022-11-23T01:35:13.7878079Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2022-11-23T01:35:13.7879901Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2022-11-23T01:35:13.7983882Z Successfully installed ninja-1.10.2 2022-11-23T01:35:13.9110819Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2022-11-23T01:35:13.9112758Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2022-11-23T01:35:13.9114468Z + [[ linux-focal-rocm5.2-py3.8 == *asan* ]] 2022-11-23T01:35:13.9115355Z + [[ linux-focal-rocm5.2-py3.8 == *-tsan* ]] 2022-11-23T01:35:13.9116093Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2022-11-23T01:35:13.9116781Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2022-11-23T01:35:13.9136777Z + [[ linux-focal-rocm5.2-py3.8 == *tbb* ]] 2022-11-23T01:35:13.9176414Z + [[ linux-focal-rocm5.2-py3.8 == *libtorch* ]] 2022-11-23T01:35:13.9177998Z + [[ linux-focal-rocm5.2-py3.8 == *-bazel-* ]] 2022-11-23T01:35:13.9179443Z + [[ linux-focal-rocm5.2-py3.8 == *-tsan* ]] 2022-11-23T01:35:13.9180449Z + cd test 2022-11-23T01:35:13.9181618Z + python -c 'import torch; print(torch.__config__.show())' 2022-11-23T01:35:15.4298472Z PyTorch built with: 2022-11-23T01:35:15.4299995Z - GCC 9.4 2022-11-23T01:35:15.4301151Z - C++ Version: 201402 2022-11-23T01:35:15.4303132Z - Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications 2022-11-23T01:35:15.4304809Z - Intel(R) MKL-DNN v2.7.0 (Git Hash 650085b2f3643aad05c629425983491d63b5c289) 2022-11-23T01:35:15.4305966Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2022-11-23T01:35:15.4307072Z - LAPACK is enabled (usually provided by MKL) 2022-11-23T01:35:15.4308037Z - NNPACK is enabled 2022-11-23T01:35:15.4308957Z - CPU capability usage: AVX2 2022-11-23T01:35:15.4309860Z - HIP Runtime 5.2.21151 2022-11-23T01:35:15.4310706Z - MIOpen 2.17.0 2022-11-23T01:35:15.4311520Z - Magma 2.6.1 2022-11-23T01:35:15.4320833Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=1.14.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=ON, 2022-11-23T01:35:15.4328077Z 2022-11-23T01:35:17.4206031Z + cd test 2022-11-23T01:35:17.4207346Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2022-11-23T01:35:18.8789880Z ATen/Parallel: 2022-11-23T01:35:18.8821057Z at::get_num_threads() : 32 2022-11-23T01:35:18.8821408Z at::get_num_interop_threads() : 32 2022-11-23T01:35:18.8821745Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2022-11-23T01:35:18.8822019Z omp_get_max_threads() : 32 2022-11-23T01:35:18.8822729Z Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications 2022-11-23T01:35:18.8823118Z mkl_get_max_threads() : 32 2022-11-23T01:35:18.8823535Z Intel(R) MKL-DNN v2.7.0 (Git Hash 650085b2f3643aad05c629425983491d63b5c289) 2022-11-23T01:35:18.8823868Z std::thread::hardware_concurrency() : 64 2022-11-23T01:35:18.8824137Z Environment variables: 2022-11-23T01:35:18.8824728Z OMP_NUM_THREADS : [not set] 2022-11-23T01:35:18.8824998Z MKL_NUM_THREADS : [not set] 2022-11-23T01:35:18.8825265Z ATen parallel backend: OpenMP 2022-11-23T01:35:18.8825432Z 2022-11-23T01:35:20.9312946Z + [[ distributed == *backward* ]] 2022-11-23T01:35:20.9314090Z + [[ distributed == *xla* ]] 2022-11-23T01:35:20.9315002Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2022-11-23T01:35:20.9316511Z + [[ linux-focal-rocm5.2-py3.8 == *libtorch* ]] 2022-11-23T01:35:20.9317320Z + [[ distributed == distributed ]] 2022-11-23T01:35:20.9318005Z + install_filelock 2022-11-23T01:35:20.9318651Z + pip_install filelock 2022-11-23T01:35:20.9319619Z + pip install --progress-bar off filelock 2022-11-23T01:35:21.4987437Z Collecting filelock 2022-11-23T01:35:21.5472675Z Downloading filelock-3.8.0-py3-none-any.whl (10 kB) 2022-11-23T01:35:22.3915225Z Installing collected packages: filelock 2022-11-23T01:35:22.4240650Z Successfully installed filelock-3.8.0 2022-11-23T01:35:22.5323111Z + install_triton 2022-11-23T01:35:22.5323970Z + local commit 2022-11-23T01:35:22.5324751Z + [[ distributed == *rocm* ]] 2022-11-23T01:35:22.5330696Z ++ get_pinned_commit triton 2022-11-23T01:35:22.5331670Z ++ cat .github/ci_commit_pins/triton.txt 2022-11-23T01:35:22.5360732Z + commit=0d7e7532279e45672555e344646f5c19c3972331 2022-11-23T01:35:22.5362809Z + pip_install --user git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python 2022-11-23T01:35:22.5364728Z + pip install --progress-bar off --user git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python 2022-11-23T01:35:23.0025748Z Collecting git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python 2022-11-23T01:35:23.0028785Z Cloning https://github.com/openai/triton (to revision 0d7e7532279e45672555e344646f5c19c3972331) to /tmp/pip-req-build-lq5e_906 2022-11-23T01:35:23.0095620Z Running command git clone --filter=blob:none --quiet https://github.com/openai/triton /tmp/pip-req-build-lq5e_906 2022-11-23T01:35:25.0950435Z Running command git rev-parse -q --verify 'sha^0d7e7532279e45672555e344646f5c19c3972331' 2022-11-23T01:35:25.1018185Z Running command git fetch -q https://github.com/openai/triton 0d7e7532279e45672555e344646f5c19c3972331 2022-11-23T01:35:25.7981748Z Running command git checkout -q 0d7e7532279e45672555e344646f5c19c3972331 2022-11-23T01:35:26.3375017Z Resolved https://github.com/openai/triton to commit 0d7e7532279e45672555e344646f5c19c3972331 2022-11-23T01:35:26.3378802Z Running command git submodule update --init --recursive -q 2022-11-23T01:35:27.4433423Z Preparing metadata (setup.py) ... [?25l- done 2022-11-23T01:35:27.6933636Z [?25hCollecting cmake 2022-11-23T01:35:27.7420984Z Downloading cmake-3.25.0-py2.py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (23.7 MB) 2022-11-23T01:35:28.2177761Z Requirement already satisfied: filelock in /opt/conda/lib/python3.8/site-packages (from triton==2.0.0) (3.8.0) 2022-11-23T01:35:28.2179389Z Requirement already satisfied: torch in /opt/conda/lib/python3.8/site-packages (from triton==2.0.0) (1.14.0a0+git1cfd385) 2022-11-23T01:35:28.2426568Z Requirement already satisfied: networkx in /opt/conda/lib/python3.8/site-packages (from torch->triton==2.0.0) (2.6.3) 2022-11-23T01:35:28.2427913Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.8/site-packages (from torch->triton==2.0.0) (4.4.0) 2022-11-23T01:35:28.2430712Z Requirement already satisfied: sympy in /opt/conda/lib/python3.8/site-packages (from torch->triton==2.0.0) (1.11.1) 2022-11-23T01:35:28.2634107Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.8/site-packages (from sympy->torch->triton==2.0.0) (1.2.1) 2022-11-23T01:35:28.2709510Z Building wheels for collected packages: triton 2022-11-23T01:36:41.3635285Z Building wheel for triton (setup.py) ... [?25l- \ | / - \ | / - \ | / done 2022-11-23T01:36:41.3844632Z [?25h Created wheel for triton: filename=triton-2.0.0-cp38-cp38-linux_x86_64.whl size=15414539 sha256=de7c381edf16cccf032054351c57b0b9462ca4370455974a8f60360b5b6fab65 2022-11-23T01:36:41.3847457Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/c0/c0/56/bdb2859a55c7764d4e97889d26a8a05b683ef97fe9b1aa7dec 2022-11-23T01:36:41.3870517Z Successfully built triton 2022-11-23T01:36:42.2688038Z Installing collected packages: cmake, triton 2022-11-23T01:36:43.4664646Z Successfully installed cmake-3.25.0 triton-2.0.0 2022-11-23T01:36:43.5814562Z + pip_install --user jinja2 2022-11-23T01:36:43.5815652Z + pip install --progress-bar off --user jinja2 2022-11-23T01:36:44.1059303Z Collecting jinja2 2022-11-23T01:36:44.1625747Z Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB) 2022-11-23T01:36:44.1877510Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.8/site-packages (from jinja2) (2.1.1) 2022-11-23T01:36:45.0399814Z Installing collected packages: jinja2 2022-11-23T01:36:45.1234868Z Successfully installed jinja2-3.1.2 2022-11-23T01:36:45.2355196Z + test_distributed 2022-11-23T01:36:45.2356538Z + echo 'Testing distributed python tests' 2022-11-23T01:36:45.2357587Z Testing distributed python tests 2022-11-23T01:36:45.2358842Z + python test/run_test.py --distributed-tests --shard 1 2 --verbose 2022-11-23T01:36:47.8862948Z Ignoring disabled issues: [] 2022-11-23T01:36:47.9071432Z Excluding distributed/rpc/test_faulty_agent on ROCm 2022-11-23T01:36:47.9072984Z Excluding distributed/rpc/test_tensorpipe_agent on ROCm 2022-11-23T01:36:47.9074262Z Excluding distributed/rpc/test_share_memory on ROCm 2022-11-23T01:36:47.9075587Z Excluding distributed/rpc/cuda/test_tensorpipe_agent on ROCm 2022-11-23T01:36:47.9077015Z Excluding distributed/_shard/sharding_plan/test_sharding_plan on ROCm 2022-11-23T01:36:47.9078894Z Excluding distributed/_shard/sharded_tensor/test_megatron_prototype on ROCm 2022-11-23T01:36:47.9080480Z Excluding distributed/_shard/sharded_tensor/test_sharded_tensor on ROCm 2022-11-23T01:36:47.9081950Z Excluding distributed/_shard/sharded_tensor/test_sharded_tensor_reshard on ROCm 2022-11-23T01:36:47.9083322Z Excluding distributed/_shard/sharded_tensor/ops/test_chunk on ROCm 2022-11-23T01:36:47.9084713Z Excluding distributed/_shard/sharded_tensor/ops/test_elementwise_ops on ROCm 2022-11-23T01:36:47.9086316Z Excluding distributed/_shard/sharded_tensor/ops/test_embedding on ROCm 2022-11-23T01:36:47.9088079Z Excluding distributed/_shard/sharded_tensor/ops/test_embedding_bag on ROCm 2022-11-23T01:36:47.9089642Z Excluding distributed/_shard/sharded_tensor/ops/test_binary_cmp on ROCm 2022-11-23T01:36:47.9090974Z Excluding distributed/_shard/sharded_tensor/ops/test_init on ROCm 2022-11-23T01:36:47.9092058Z Excluding distributed/_shard/sharded_tensor/ops/test_linear on ROCm 2022-11-23T01:36:47.9093081Z Excluding distributed/_shard/sharded_tensor/ops/test_math_ops on ROCm 2022-11-23T01:36:47.9094101Z Excluding distributed/_shard/sharded_tensor/ops/test_matrix_ops on ROCm 2022-11-23T01:36:47.9095153Z Excluding distributed/_shard/sharded_tensor/ops/test_softmax on ROCm 2022-11-23T01:36:47.9096727Z Excluding distributed/_shard/sharded_optim/test_sharded_optim on ROCm 2022-11-23T01:36:47.9097717Z Excluding distributed/_shard/test_partial_tensor on ROCm 2022-11-23T01:36:47.9098684Z Excluding distributed/_shard/test_replicated_tensor on ROCm 2022-11-23T01:36:47.9160644Z ##[warning] Gathered no stats from artifacts. Proceeding with default sharding plan. 2022-11-23T01:36:47.9162344Z Selected tests: 2022-11-23T01:36:47.9162925Z distributed/algorithms/quantization/test_quantization 2022-11-23T01:36:47.9163544Z distributed/test_distributed_spawn 2022-11-23T01:36:47.9164084Z distributed/test_pg_wrapper 2022-11-23T01:36:47.9164628Z distributed/test_multi_threaded_pg 2022-11-23T01:36:47.9165187Z distributed/test_dynamo_distributed 2022-11-23T01:36:47.9165718Z distributed/test_c10d_spawn_ucc 2022-11-23T01:36:47.9166256Z distributed/test_c10d_spawn_gloo 2022-11-23T01:36:47.9166809Z distributed/test_c10d_object_collectives 2022-11-23T01:36:47.9167905Z distributed/test_c10d_gloo 2022-11-23T01:36:47.9168438Z distributed/test_c10d_common 2022-11-23T01:36:47.9168993Z distributed/pipeline/sync/test_transparency 2022-11-23T01:36:47.9169601Z distributed/pipeline/sync/test_pipeline 2022-11-23T01:36:47.9170175Z distributed/pipeline/sync/test_phony 2022-11-23T01:36:47.9170754Z distributed/pipeline/sync/test_inplace 2022-11-23T01:36:47.9171380Z distributed/pipeline/sync/test_deferred_batch_norm 2022-11-23T01:36:47.9172012Z distributed/pipeline/sync/test_checkpoint 2022-11-23T01:36:47.9172580Z distributed/pipeline/sync/test_balance 2022-11-23T01:36:47.9173167Z distributed/pipeline/sync/skip/test_tracker 2022-11-23T01:36:47.9173773Z distributed/pipeline/sync/skip/test_portal 2022-11-23T01:36:47.9174420Z distributed/pipeline/sync/skip/test_inspect_skip_layout 2022-11-23T01:36:47.9175047Z distributed/pipeline/sync/skip/test_api 2022-11-23T01:36:47.9175669Z distributed/optim/test_apply_optimizer_in_backward 2022-11-23T01:36:47.9176226Z distributed/fsdp/test_wrap 2022-11-23T01:36:47.9176774Z distributed/fsdp/test_shard_utils 2022-11-23T01:36:47.9177314Z distributed/fsdp/test_fsdp_uneven 2022-11-23T01:36:47.9177883Z distributed/fsdp/test_fsdp_tp_integration 2022-11-23T01:36:47.9178450Z distributed/fsdp/test_fsdp_state_dict 2022-11-23T01:36:47.9179012Z distributed/fsdp/test_fsdp_pure_fp16 2022-11-23T01:36:47.9179559Z distributed/fsdp/test_fsdp_optim_state 2022-11-23T01:36:47.9180145Z distributed/fsdp/test_fsdp_multiple_forward 2022-11-23T01:36:47.9180713Z distributed/fsdp/test_fsdp_misc 2022-11-23T01:36:47.9181258Z distributed/fsdp/test_fsdp_memory 2022-11-23T01:36:47.9181836Z distributed/fsdp/test_fsdp_ignored_modules 2022-11-23T01:36:47.9182382Z distributed/fsdp/test_fsdp_fx 2022-11-23T01:36:47.9182942Z distributed/fsdp/test_fsdp_flatten_params 2022-11-23T01:36:47.9183506Z distributed/fsdp/test_fsdp_core 2022-11-23T01:36:47.9184039Z distributed/fsdp/test_fsdp_comm 2022-11-23T01:36:47.9184582Z distributed/fsdp/test_fsdp_checkpoint 2022-11-23T01:36:47.9185189Z distributed/fsdp/test_distributed_checkpoint 2022-11-23T01:36:47.9185757Z distributed/elastic/utils/util_test 2022-11-23T01:36:47.9186336Z distributed/elastic/utils/distributed_test 2022-11-23T01:36:47.9186948Z distributed/elastic/timer/local_timer_example 2022-11-23T01:36:47.9187568Z distributed/elastic/multiprocessing/api_test 2022-11-23T01:36:47.9188150Z distributed/elastic/events/lib_test 2022-11-23T01:36:47.9188716Z distributed/checkpoint/test_traverse 2022-11-23T01:36:47.9189332Z distributed/checkpoint/test_file_system_checkpoint_cpu 2022-11-23T01:36:47.9189969Z distributed/checkpoint/test_dedup_tensors 2022-11-23T01:36:47.9190532Z distributed/algorithms/test_join 2022-11-23T01:36:47.9191068Z distributed/_tensor/test_view_ops 2022-11-23T01:36:47.9191679Z distributed/_tensor/test_tensor_ops 2022-11-23T01:36:47.9192243Z distributed/_tensor/test_pointwise_ops 2022-11-23T01:36:47.9192785Z distributed/_tensor/test_math_ops 2022-11-23T01:36:47.9193323Z distributed/_tensor/test_device_mesh 2022-11-23T01:36:47.9194060Z distributed/_tensor/test_api 2022-11-23T01:36:47.9194615Z distributed/_tensor/parallel/test_tp_style 2022-11-23T01:36:47.9195237Z distributed/_tensor/parallel/test_parallelize_api 2022-11-23T01:36:47.9195805Z distributed/_shard/test_sharder 2022-11-23T01:36:47.9196405Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 2022-11-23T01:36:47.9197017Z distributed/_composable/test_fully_shard 2022-11-23T01:36:47.9197600Z distributed/_composable/test_checkpoint 2022-11-23T01:36:47.9223968Z Prioritized test from test file changes. 2022-11-23T01:36:47.9224572Z reordering tests for PR: 2022-11-23T01:36:47.9225057Z prioritized: [] 2022-11-23T01:36:47.9236257Z the rest: ['distributed/algorithms/quantization/test_quantization', 'distributed/test_distributed_spawn', 'distributed/test_pg_wrapper', 'distributed/test_multi_threaded_pg', 'distributed/test_dynamo_distributed', 'distributed/test_c10d_spawn_ucc', 'distributed/test_c10d_spawn_gloo', 'distributed/test_c10d_object_collectives', 'distributed/test_c10d_gloo', 'distributed/test_c10d_common', 'distributed/pipeline/sync/test_transparency', 'distributed/pipeline/sync/test_pipeline', 'distributed/pipeline/sync/test_phony', 'distributed/pipeline/sync/test_inplace', 'distributed/pipeline/sync/test_deferred_batch_norm', 'distributed/pipeline/sync/test_checkpoint', 'distributed/pipeline/sync/test_balance', 'distributed/pipeline/sync/skip/test_tracker', 'distributed/pipeline/sync/skip/test_portal', 'distributed/pipeline/sync/skip/test_inspect_skip_layout', 'distributed/pipeline/sync/skip/test_api', 'distributed/optim/test_apply_optimizer_in_backward', 'distributed/fsdp/test_wrap', 'distributed/fsdp/test_shard_utils', 'distributed/fsdp/test_fsdp_uneven', 'distributed/fsdp/test_fsdp_tp_integration', 'distributed/fsdp/test_fsdp_state_dict', 'distributed/fsdp/test_fsdp_pure_fp16', 'distributed/fsdp/test_fsdp_optim_state', 'distributed/fsdp/test_fsdp_multiple_forward', 'distributed/fsdp/test_fsdp_misc', 'distributed/fsdp/test_fsdp_memory', 'distributed/fsdp/test_fsdp_ignored_modules', 'distributed/fsdp/test_fsdp_fx', 'distributed/fsdp/test_fsdp_flatten_params', 'distributed/fsdp/test_fsdp_core', 'distributed/fsdp/test_fsdp_comm', 'distributed/fsdp/test_fsdp_checkpoint', 'distributed/fsdp/test_distributed_checkpoint', 'distributed/elastic/utils/util_test', 'distributed/elastic/utils/distributed_test', 'distributed/elastic/timer/local_timer_example', 'distributed/elastic/multiprocessing/api_test', 'distributed/elastic/events/lib_test', 'distributed/checkpoint/test_traverse', 'distributed/checkpoint/test_file_system_checkpoint_cpu', 'distributed/checkpoint/test_dedup_tensors', 'distributed/algorithms/test_join', 'distributed/_tensor/test_view_ops', 'distributed/_tensor/test_tensor_ops', 'distributed/_tensor/test_pointwise_ops', 'distributed/_tensor/test_math_ops', 'distributed/_tensor/test_device_mesh', 'distributed/_tensor/test_api', 'distributed/_tensor/parallel/test_tp_style', 'distributed/_tensor/parallel/test_parallelize_api', 'distributed/_shard/test_sharder', 'distributed/_shard/sharded_tensor/ops/test_tensor_ops', 'distributed/_composable/test_fully_shard', 'distributed/_composable/test_checkpoint'] 2022-11-23T01:36:47.9242788Z 2022-11-23T01:36:47.9243785Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/pytorch/test/.pytorch-slow-tests.json 2022-11-23T01:36:47.9616633Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2022-11-23T01:36:47.9953298Z parallel (file granularity) tests: 2022-11-23T01:36:47.9954038Z 2022-11-23T01:36:47.9954653Z serial (file granularity) tests: 2022-11-23T01:36:47.9955524Z distributed/algorithms/quantization/test_quantization 2022-11-23T01:36:47.9956381Z distributed/test_distributed_spawn 2022-11-23T01:36:47.9957108Z distributed/test_pg_wrapper 2022-11-23T01:36:47.9957853Z distributed/test_multi_threaded_pg 2022-11-23T01:36:47.9959143Z distributed/test_dynamo_distributed 2022-11-23T01:36:47.9959894Z distributed/test_c10d_spawn_ucc 2022-11-23T01:36:47.9960628Z distributed/test_c10d_spawn_gloo 2022-11-23T01:36:47.9961392Z distributed/test_c10d_object_collectives 2022-11-23T01:36:47.9962128Z distributed/test_c10d_gloo 2022-11-23T01:36:47.9962806Z distributed/test_c10d_common 2022-11-23T01:36:47.9963584Z distributed/pipeline/sync/test_transparency 2022-11-23T01:36:47.9964423Z distributed/pipeline/sync/test_pipeline 2022-11-23T01:36:47.9965228Z distributed/pipeline/sync/test_phony 2022-11-23T01:36:47.9966027Z distributed/pipeline/sync/test_inplace 2022-11-23T01:36:47.9966882Z distributed/pipeline/sync/test_deferred_batch_norm 2022-11-23T01:36:47.9967927Z distributed/pipeline/sync/test_checkpoint 2022-11-23T01:36:47.9968733Z distributed/pipeline/sync/test_balance 2022-11-23T01:36:47.9969548Z distributed/pipeline/sync/skip/test_tracker 2022-11-23T01:36:47.9970390Z distributed/pipeline/sync/skip/test_portal 2022-11-23T01:36:47.9971468Z distributed/pipeline/sync/skip/test_inspect_skip_layout 2022-11-23T01:36:47.9972339Z distributed/pipeline/sync/skip/test_api 2022-11-23T01:36:47.9973199Z distributed/optim/test_apply_optimizer_in_backward 2022-11-23T01:36:47.9973988Z distributed/fsdp/test_wrap 2022-11-23T01:36:47.9974718Z distributed/fsdp/test_shard_utils 2022-11-23T01:36:47.9975462Z distributed/fsdp/test_fsdp_uneven 2022-11-23T01:36:47.9976248Z distributed/fsdp/test_fsdp_tp_integration 2022-11-23T01:36:47.9977022Z distributed/fsdp/test_fsdp_state_dict 2022-11-23T01:36:47.9977801Z distributed/fsdp/test_fsdp_pure_fp16 2022-11-23T01:36:47.9978574Z distributed/fsdp/test_fsdp_optim_state 2022-11-23T01:36:47.9979397Z distributed/fsdp/test_fsdp_multiple_forward 2022-11-23T01:36:47.9980181Z distributed/fsdp/test_fsdp_misc 2022-11-23T01:36:47.9980923Z distributed/fsdp/test_fsdp_memory 2022-11-23T01:36:47.9981701Z distributed/fsdp/test_fsdp_ignored_modules 2022-11-23T01:36:47.9982469Z distributed/fsdp/test_fsdp_fx 2022-11-23T01:36:47.9983264Z distributed/fsdp/test_fsdp_flatten_params 2022-11-23T01:36:47.9984038Z distributed/fsdp/test_fsdp_core 2022-11-23T01:36:47.9984769Z distributed/fsdp/test_fsdp_comm 2022-11-23T01:36:47.9985514Z distributed/fsdp/test_fsdp_checkpoint 2022-11-23T01:36:47.9986347Z distributed/fsdp/test_distributed_checkpoint 2022-11-23T01:36:47.9987152Z distributed/elastic/utils/util_test 2022-11-23T01:36:47.9987953Z distributed/elastic/utils/distributed_test 2022-11-23T01:36:47.9988796Z distributed/elastic/timer/local_timer_example 2022-11-23T01:36:47.9989653Z distributed/elastic/multiprocessing/api_test 2022-11-23T01:36:47.9990442Z distributed/elastic/events/lib_test 2022-11-23T01:36:47.9991215Z distributed/checkpoint/test_traverse 2022-11-23T01:36:47.9992080Z distributed/checkpoint/test_file_system_checkpoint_cpu 2022-11-23T01:36:47.9992957Z distributed/checkpoint/test_dedup_tensors 2022-11-23T01:36:47.9993743Z distributed/algorithms/test_join 2022-11-23T01:36:47.9994505Z distributed/_tensor/test_view_ops 2022-11-23T01:36:47.9995237Z distributed/_tensor/test_tensor_ops 2022-11-23T01:36:47.9996005Z distributed/_tensor/test_pointwise_ops 2022-11-23T01:36:47.9996774Z distributed/_tensor/test_math_ops 2022-11-23T01:36:47.9997515Z distributed/_tensor/test_device_mesh 2022-11-23T01:36:47.9998256Z distributed/_tensor/test_api 2022-11-23T01:36:47.9999028Z distributed/_tensor/parallel/test_tp_style 2022-11-23T01:36:47.9999879Z distributed/_tensor/parallel/test_parallelize_api 2022-11-23T01:36:48.0000689Z distributed/_shard/test_sharder 2022-11-23T01:36:48.0001530Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 2022-11-23T01:36:48.0002386Z distributed/_composable/test_fully_shard 2022-11-23T01:36:48.0003189Z distributed/_composable/test_checkpoint 2022-11-23T01:36:50.3038719Z Ignoring disabled issues: [] 2022-11-23T01:36:52.4432904Z Running distributed/algorithms/quantization/test_quantization ... [2022-11-23 01:36:52.442448] 2022-11-23T01:36:52.4524429Z MPI not available -- MPI backend tests will be skipped 2022-11-23T01:36:52.4526536Z Map different backends to different shards for distributed/algorithms/quantization/test_quantization: {'gloo': 1, 'nccl': 2} 2022-11-23T01:36:52.4531426Z Running distributed tests for the test backend with env init_method in shard 1 of 2 2022-11-23T01:36:52.4544237Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:36:52.453656] 2022-11-23T01:36:56.3505101Z 2022-11-23T01:36:56.3506332Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization 2022-11-23T01:36:56.3520375Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/pytorch/test/test-reports/distributed-algorithms-quantization-test_quantization_1o95_6v9) 2022-11-23T01:36:56.3521976Z 2022-11-23T01:36:56.3522490Z 2022-11-23T01:36:56.3523766Z ##[endgroup] 2022-11-23T01:36:56.3525978Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/pytorch/test/test-reports/distributed-algorithms-quantization-test_quantization_1o95_6v9) 2022-11-23T01:36:56.3527086Z 2022-11-23T01:36:56.3527644Z Running distributed tests for the test backend with file init_method in shard 1 of 2 2022-11-23T01:36:56.3535747Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:36:56.353040] 2022-11-23T01:37:00.6841006Z 2022-11-23T01:37:00.6842224Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization 2022-11-23T01:37:00.6845687Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/pytorch/test/test-reports/distributed-algorithms-quantization-test_quantization_g9cjmxuq) 2022-11-23T01:37:00.6848201Z 2022-11-23T01:37:00.6848931Z 2022-11-23T01:37:00.6849959Z ##[endgroup] 2022-11-23T01:37:00.6853135Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/pytorch/test/test-reports/distributed-algorithms-quantization-test_quantization_g9cjmxuq) 2022-11-23T01:37:00.6854783Z 2022-11-23T01:37:00.6864338Z Shard 1: nccl should be run in 2 2022-11-23T01:37:00.6865352Z Running distributed tests for the gloo backend with env init_method in shard 1 of 2 2022-11-23T01:37:00.6876306Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:37:00.686868] 2022-11-23T01:37:37.6401008Z 2022-11-23T01:37:37.6402057Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization 2022-11-23T01:37:37.6404517Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/pytorch/test/test-reports/distributed-algorithms-quantization-test_quantization_i0rcc1f8) 2022-11-23T01:37:37.6407383Z , <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]> 2022-11-23T01:37:37.6410003Z test_all_gather_bfp16 (__main__.DistQuantizationTests) 2022-11-23T01:37:37.6411080Z test_all_gather_fp16 (__main__.DistQuantizationTests) 2022-11-23T01:37:37.6411995Z test_all_to_all_bfp16 (__main__.DistQuantizationTests) 2022-11-23T01:37:37.6413046Z test_all_to_all_fp16 (__main__.DistQuantizationTests) 2022-11-23T01:37:37.6414520Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) 2022-11-23T01:37:37.6416249Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests) 2022-11-23T01:37:37.6417544Z 2022-11-23T01:37:37.6420500Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2022-11-23T01:37:37.6422965Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:37.6424180Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:37.6425744Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:37.6426944Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:37.6427668Z 2022-11-23T01:37:37.6428026Z Running tests... 2022-11-23T01:37:37.6429619Z ---------------------------------------------------------------------- 2022-11-23T01:37:37.6431807Z test_all_gather_bfp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1235 2022-11-23T01:37:37.6433424Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1236 2022-11-23T01:37:37.6435210Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T01:37:37.6437642Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:37.6439096Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:37.6440816Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:37.6442141Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:37.6443340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:37:37.6445169Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:37.6446433Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:37.6448462Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:37.6449693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:37.6450922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:37:37.6452156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:37:37.6453901Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:37:37.6455253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:37:37.6456968Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:37:37.6457964Z ok (5.422s) 2022-11-23T01:37:37.6458319Z 2022-11-23T01:37:37.6459029Z ---------------------------------------------------------------------- 2022-11-23T01:37:37.6459845Z Ran 1 test in 5.423s 2022-11-23T01:37:37.6460241Z 2022-11-23T01:37:37.6460453Z OK 2022-11-23T01:37:37.6460769Z 2022-11-23T01:37:37.6461058Z Generating XML reports... 2022-11-23T01:37:37.6462802Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013704.xml 2022-11-23T01:37:37.6464752Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2022-11-23T01:37:37.6466511Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:37.6467646Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:37.6469450Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:37.6470640Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:37.6471200Z 2022-11-23T01:37:37.6471450Z Running tests... 2022-11-23T01:37:37.6472515Z ---------------------------------------------------------------------- 2022-11-23T01:37:37.6473803Z test_all_gather_fp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1442 2022-11-23T01:37:37.6475119Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1443 2022-11-23T01:37:37.6476383Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T01:37:37.6478095Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:37.6479394Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:37.6480944Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:37.6482138Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:37.6483237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:37:37.6484851Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:37.6485990Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:37.6487513Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:37.6489014Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:37.6490129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:37:37.6491355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:37:37.6492578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:37:37.6494264Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:37:37.6496046Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:37:37.6497029Z ok (4.646s) 2022-11-23T01:37:37.6497387Z 2022-11-23T01:37:37.6498097Z ---------------------------------------------------------------------- 2022-11-23T01:37:37.6498965Z Ran 1 test in 4.647s 2022-11-23T01:37:37.6499375Z 2022-11-23T01:37:37.6499597Z OK 2022-11-23T01:37:37.6499934Z 2022-11-23T01:37:37.6500244Z Generating XML reports... 2022-11-23T01:37:37.6502057Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013713.xml 2022-11-23T01:37:37.6504142Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2022-11-23T01:37:37.6506002Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:37.6507206Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:37.6508803Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:37.6510049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:37.6510641Z 2022-11-23T01:37:37.6510906Z Running tests... 2022-11-23T01:37:37.6512043Z ---------------------------------------------------------------------- 2022-11-23T01:37:37.6513285Z test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_fp16 (0.001s) 2022-11-23T01:37:37.6514210Z 2022-11-23T01:37:37.6514961Z ---------------------------------------------------------------------- 2022-11-23T01:37:37.6515817Z Ran 1 test in 0.001s 2022-11-23T01:37:37.6516221Z 2022-11-23T01:37:37.6516477Z OK (skipped=1) 2022-11-23T01:37:37.6516867Z 2022-11-23T01:37:37.6517171Z Generating XML reports... 2022-11-23T01:37:37.6518978Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013722.xml 2022-11-23T01:37:37.6521059Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2022-11-23T01:37:37.6522890Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:37.6524094Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:37.6525849Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:37.6527129Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:37.6528051Z 2022-11-23T01:37:37.6528327Z Running tests... 2022-11-23T01:37:37.6529479Z ---------------------------------------------------------------------- 2022-11-23T01:37:37.6530723Z test_all_to_all_fp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_fp16 (0.001s) 2022-11-23T01:37:37.6531454Z 2022-11-23T01:37:37.6532342Z ---------------------------------------------------------------------- 2022-11-23T01:37:37.6533184Z Ran 1 test in 0.001s 2022-11-23T01:37:37.6533593Z 2022-11-23T01:37:37.6533856Z OK (skipped=1) 2022-11-23T01:37:37.6534240Z 2022-11-23T01:37:37.6534548Z Generating XML reports... 2022-11-23T01:37:37.6536373Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013725.xml 2022-11-23T01:37:37.6537936Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2022-11-23T01:37:37.6538725Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:37.6539250Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:37.6539922Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:37.6540456Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:37.6540718Z 2022-11-23T01:37:37.6540840Z Running tests... 2022-11-23T01:37:37.6541320Z ---------------------------------------------------------------------- 2022-11-23T01:37:37.6541887Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_single_bfp16 (0.001s) 2022-11-23T01:37:37.6542230Z 2022-11-23T01:37:37.6542550Z ---------------------------------------------------------------------- 2022-11-23T01:37:37.6542927Z Ran 1 test in 0.001s 2022-11-23T01:37:37.6543111Z 2022-11-23T01:37:37.6543212Z OK (skipped=1) 2022-11-23T01:37:37.6543382Z 2022-11-23T01:37:37.6543516Z Generating XML reports... 2022-11-23T01:37:37.6544282Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013729.xml 2022-11-23T01:37:37.6545170Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2022-11-23T01:37:37.6545966Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:37.6546482Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:37.6547175Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:37.6547704Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:37.6548056Z 2022-11-23T01:37:37.6548183Z Running tests... 2022-11-23T01:37:37.6548647Z ---------------------------------------------------------------------- 2022-11-23T01:37:37.6549126Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_single_fp16 (0.001s) 2022-11-23T01:37:37.6549405Z 2022-11-23T01:37:37.6549668Z ---------------------------------------------------------------------- 2022-11-23T01:37:37.6549974Z Ran 1 test in 0.001s 2022-11-23T01:37:37.6550122Z 2022-11-23T01:37:37.6550217Z OK (skipped=1) 2022-11-23T01:37:37.6550361Z 2022-11-23T01:37:37.6550471Z Generating XML reports... 2022-11-23T01:37:37.6551101Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013733.xml 2022-11-23T01:37:37.6551473Z 2022-11-23T01:37:37.6551812Z ##[endgroup] 2022-11-23T01:37:37.6552570Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/pytorch/test/test-reports/distributed-algorithms-quantization-test_quantization_i0rcc1f8) 2022-11-23T01:37:37.6552982Z 2022-11-23T01:37:37.6553180Z Running distributed tests for the gloo backend with file init_method in shard 1 of 2 2022-11-23T01:37:37.6553975Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:37:37.642813] 2022-11-23T01:38:13.5655643Z 2022-11-23T01:38:13.5656736Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization 2022-11-23T01:38:13.5659092Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/pytorch/test/test-reports/distributed-algorithms-quantization-test_quantization_cv8nk109) 2022-11-23T01:38:13.5662114Z , <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]> 2022-11-23T01:38:13.5664896Z test_all_gather_bfp16 (__main__.DistQuantizationTests) 2022-11-23T01:38:13.5665890Z test_all_gather_fp16 (__main__.DistQuantizationTests) 2022-11-23T01:38:13.5667251Z test_all_to_all_bfp16 (__main__.DistQuantizationTests) 2022-11-23T01:38:13.5668492Z test_all_to_all_fp16 (__main__.DistQuantizationTests) 2022-11-23T01:38:13.5669836Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) 2022-11-23T01:38:13.5670808Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests) 2022-11-23T01:38:13.5671681Z 2022-11-23T01:38:13.5673486Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2022-11-23T01:38:13.5675336Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:13.5676550Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:13.5678388Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:13.5679607Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:13.5680184Z 2022-11-23T01:38:13.5680436Z Running tests... 2022-11-23T01:38:13.5681531Z ---------------------------------------------------------------------- 2022-11-23T01:38:13.5682860Z test_all_gather_bfp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1979 2022-11-23T01:38:13.5684419Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1980 2022-11-23T01:38:13.5686541Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T01:38:13.5688684Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:13.5689828Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:13.5691372Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:13.5692562Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:13.5694012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:38:13.5696268Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:13.5697821Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:13.5699842Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:13.5701066Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:13.5702177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:38:13.5703410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:38:13.5704626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:38:13.5706366Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:38:13.5708179Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:38:13.5709179Z ok (4.675s) 2022-11-23T01:38:13.5709534Z 2022-11-23T01:38:13.5710238Z ---------------------------------------------------------------------- 2022-11-23T01:38:13.5711073Z Ran 1 test in 4.675s 2022-11-23T01:38:13.5711470Z 2022-11-23T01:38:13.5711684Z OK 2022-11-23T01:38:13.5711984Z 2022-11-23T01:38:13.5712279Z Generating XML reports... 2022-11-23T01:38:13.5713992Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013741.xml 2022-11-23T01:38:13.5715953Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2022-11-23T01:38:13.5717715Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:13.5718863Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:13.5720383Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:13.5721580Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:13.5722155Z 2022-11-23T01:38:13.5722401Z Running tests... 2022-11-23T01:38:13.5723482Z ---------------------------------------------------------------------- 2022-11-23T01:38:13.5724792Z test_all_gather_fp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2186 2022-11-23T01:38:13.5726113Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2187 2022-11-23T01:38:13.5727387Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T01:38:13.5729315Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:13.5730451Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:13.5731976Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:13.5733165Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:13.5734502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:38:13.5736157Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:13.5737288Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:13.5738811Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:13.5739966Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:13.5740903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:38:13.5741928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:38:13.5742985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:38:13.5744553Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:38:13.5746085Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:38:13.5746921Z ok (4.604s) 2022-11-23T01:38:13.5747223Z 2022-11-23T01:38:13.5747823Z ---------------------------------------------------------------------- 2022-11-23T01:38:13.5748513Z Ran 1 test in 4.604s 2022-11-23T01:38:13.5748846Z 2022-11-23T01:38:13.5749011Z OK 2022-11-23T01:38:13.5749282Z 2022-11-23T01:38:13.5749530Z Generating XML reports... 2022-11-23T01:38:13.5750979Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013749.xml 2022-11-23T01:38:13.5752927Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2022-11-23T01:38:13.5754696Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:13.5755837Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:13.5757350Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:13.5758538Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:13.5759081Z 2022-11-23T01:38:13.5759323Z Running tests... 2022-11-23T01:38:13.5760221Z ---------------------------------------------------------------------- 2022-11-23T01:38:13.5761227Z test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_fp16 (0.001s) 2022-11-23T01:38:13.5761823Z 2022-11-23T01:38:13.5762415Z ---------------------------------------------------------------------- 2022-11-23T01:38:13.5763106Z Ran 1 test in 0.001s 2022-11-23T01:38:13.5763433Z 2022-11-23T01:38:13.5763650Z OK (skipped=1) 2022-11-23T01:38:13.5763969Z 2022-11-23T01:38:13.5764218Z Generating XML reports... 2022-11-23T01:38:13.5765637Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013758.xml 2022-11-23T01:38:13.5767299Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2022-11-23T01:38:13.5769183Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:13.5770323Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:13.5771844Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:13.5773028Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:13.5773691Z 2022-11-23T01:38:13.5773939Z Running tests... 2022-11-23T01:38:13.5775017Z ---------------------------------------------------------------------- 2022-11-23T01:38:13.5776401Z test_all_to_all_fp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_fp16 (0.001s) 2022-11-23T01:38:13.5777105Z 2022-11-23T01:38:13.5777809Z ---------------------------------------------------------------------- 2022-11-23T01:38:13.5778615Z Ran 1 test in 0.001s 2022-11-23T01:38:13.5779001Z 2022-11-23T01:38:13.5779251Z OK (skipped=1) 2022-11-23T01:38:13.5779487Z 2022-11-23T01:38:13.5779630Z Generating XML reports... 2022-11-23T01:38:13.5780320Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013801.xml 2022-11-23T01:38:13.5781060Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2022-11-23T01:38:13.5781720Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:13.5782201Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:13.5782782Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:13.5783232Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:13.5783446Z 2022-11-23T01:38:13.5783543Z Running tests... 2022-11-23T01:38:13.5783945Z ---------------------------------------------------------------------- 2022-11-23T01:38:13.5784424Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_single_bfp16 (0.001s) 2022-11-23T01:38:13.5784712Z 2022-11-23T01:38:13.5784980Z ---------------------------------------------------------------------- 2022-11-23T01:38:13.5785281Z Ran 1 test in 0.001s 2022-11-23T01:38:13.5785429Z 2022-11-23T01:38:13.5785525Z OK (skipped=1) 2022-11-23T01:38:13.5785666Z 2022-11-23T01:38:13.5785779Z Generating XML reports... 2022-11-23T01:38:13.5786427Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013805.xml 2022-11-23T01:38:13.5787160Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization 2022-11-23T01:38:13.5787817Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:38:13.5788249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:38:13.5788809Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:38:13.5789256Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:38:13.5789467Z 2022-11-23T01:38:13.5789562Z Running tests... 2022-11-23T01:38:13.5789966Z ---------------------------------------------------------------------- 2022-11-23T01:38:13.5790437Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_single_fp16 (0.001s) 2022-11-23T01:38:13.5790722Z 2022-11-23T01:38:13.5790984Z ---------------------------------------------------------------------- 2022-11-23T01:38:13.5791292Z Ran 1 test in 0.001s 2022-11-23T01:38:13.5791439Z 2022-11-23T01:38:13.5791533Z OK (skipped=1) 2022-11-23T01:38:13.5791666Z 2022-11-23T01:38:13.5791777Z Generating XML reports... 2022-11-23T01:38:13.5792411Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013809.xml 2022-11-23T01:38:13.5792776Z 2022-11-23T01:38:13.5793167Z ##[endgroup] 2022-11-23T01:38:13.5793871Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/pytorch/test/test-reports/distributed-algorithms-quantization-test_quantization_cv8nk109) 2022-11-23T01:38:13.5794280Z 2022-11-23T01:38:13.5794543Z Running distributed/test_distributed_spawn ... [2022-11-23 01:38:13.568458] 2022-11-23T01:38:13.5847264Z MPI not available -- MPI backend tests will be skipped 2022-11-23T01:38:13.5848199Z Map different backends to different shards for distributed/test_distributed_spawn: {'gloo': 1, 'nccl': 2, 'ucc': 1} 2022-11-23T01:38:13.5850456Z Running distributed tests for the test backend with env init_method in shard 1 of 2 2022-11-23T01:38:13.5864211Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:38:13.585763] 2022-11-23T01:38:17.6518492Z 2022-11-23T01:38:17.6519433Z Expand the folded group to see the log file of distributed/test_distributed_spawn 2022-11-23T01:38:17.6521489Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_distributed_spawn_bmoesz8n) 2022-11-23T01:38:17.6522727Z 2022-11-23T01:38:17.6523227Z 2022-11-23T01:38:17.6523929Z ##[endgroup] 2022-11-23T01:38:17.6526279Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_distributed_spawn_bmoesz8n) 2022-11-23T01:38:17.6527192Z 2022-11-23T01:38:17.6540028Z Running distributed tests for the test backend with file init_method in shard 1 of 2 2022-11-23T01:38:17.6547995Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:38:17.654293] 2022-11-23T01:38:21.9613378Z 2022-11-23T01:38:21.9614547Z Expand the folded group to see the log file of distributed/test_distributed_spawn 2022-11-23T01:38:21.9617276Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_distributed_spawn_7_x4l9vc) 2022-11-23T01:38:21.9618839Z 2022-11-23T01:38:21.9619315Z 2022-11-23T01:38:21.9620044Z ##[endgroup] 2022-11-23T01:38:21.9621950Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_distributed_spawn_7_x4l9vc) 2022-11-23T01:38:21.9622834Z 2022-11-23T01:38:21.9631864Z Shard 1: nccl should be run in 2 2022-11-23T01:38:21.9635364Z Running distributed tests for the gloo backend with env init_method in shard 1 of 2 2022-11-23T01:38:21.9645840Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:38:21.964012] 2022-11-23T02:13:37.5466294Z 2022-11-23T02:13:37.5467041Z Expand the folded group to see the log file of distributed/test_distributed_spawn 2022-11-23T02:13:37.5481968Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_distributed_spawn_n0h3_4pw) 2022-11-23T02:13:37.5483858Z 2022-11-23T02:13:37.5591313Z , <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_cat_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_stack_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_zero_output_features>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]> 2022-11-23T02:13:37.5640621Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5641443Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5642035Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5651036Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5651694Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5652400Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5653130Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5653897Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5654654Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5655468Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5656337Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5657144Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5657956Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5658724Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5659418Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5660117Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5660872Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5661525Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5662209Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5663080Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5663850Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5664588Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5665186Z test_all_gather (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5665777Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5666408Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5667029Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5667648Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5668264Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5668843Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5669513Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5670197Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5670757Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5672074Z test_all_gather_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5673296Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5674778Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5675680Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5676892Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5678168Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5679460Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5680684Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5681960Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5683211Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5684191Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5685631Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5686510Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5687366Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5688363Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5689207Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5689879Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5690758Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5691631Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5692483Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5693314Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5694118Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5694817Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5695408Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5696100Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5696902Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5697568Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5698236Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5698908Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5699714Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5700275Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5700919Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5701529Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5702166Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5702796Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5703335Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5703763Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5704266Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5704767Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5705255Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5705839Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5706380Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5706957Z test_all_to_all (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5707371Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5707846Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5708351Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5708849Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5709348Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5709839Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5710334Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5710772Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5711309Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5711836Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5712393Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5712941Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5713500Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5713983Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5714570Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5715100Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5715650Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5716270Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5716832Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5717320Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5717874Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5718426Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5718996Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5719529Z test_average_parameters (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5720022Z test_backend_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5720500Z test_backend_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5720915Z test_barrier (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5721385Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5721949Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5722450Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5722954Z test_barrier_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5723434Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5723865Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5724451Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5725002Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5725500Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5725999Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5726526Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5727111Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5727648Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5728182Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5728703Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5729245Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5729774Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5730285Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5730853Z test_broadcast (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5731343Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5731933Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5732629Z test_broadcast_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5733310Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5733901Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5734560Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5735279Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5736428Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5737657Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5738832Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5740142Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5741102Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5742404Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5743660Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5744698Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5746009Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5747357Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5748575Z test_ddp_device (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5749828Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5751122Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5752494Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5753408Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5754625Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5755967Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5757240Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5758483Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5759813Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5761266Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5762935Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5764478Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5766060Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5767618Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5769283Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5770475Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5771785Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5772606Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5773454Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5774106Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5774672Z test_ddp_inference (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5775287Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5775810Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5776396Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5777008Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5777653Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5778309Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5778979Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5779587Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5780106Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5780714Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5781338Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5781960Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5782588Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5783197Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5783958Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5784507Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5785131Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5785722Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5786316Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5787044Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5787655Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5788190Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5788843Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5789481Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5790077Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5790647Z test_destroy_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5791234Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5791843Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5792395Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5793034Z test_gather (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5793608Z test_gather_checks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5794333Z test_gather_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5794918Z test_gather_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5795478Z test_gather_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5795958Z test_gather_object (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5796534Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5797111Z test_get_backend (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5797663Z test_get_future (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5798201Z test_get_rank (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5798766Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5799285Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5799869Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5800422Z test_irecv (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5800950Z test_isend (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5801529Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5802124Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5802656Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5803299Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5804117Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5804728Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5805353Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5805981Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5806620Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5807160Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5807851Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5808475Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5809091Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5809680Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5810268Z test_new_subgroups (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5810777Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5811427Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5812130Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5812905Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5813543Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5814383Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5815099Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5815647Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5816260Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5816889Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5817524Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5818170Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5818925Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5819678Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5820290Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5820950Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5821695Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5823685Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5824816Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5826070Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5827242Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5828127Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5829351Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5830632Z test_reduce_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5831746Z test_reduce_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5832884Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5834019Z test_reduce_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5834924Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5836134Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5837262Z test_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5838400Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5839586Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5840730Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5841595Z test_scatter (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5842704Z test_scatter_checks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5843881Z test_scatter_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5896200Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5897721Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5898855Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5899877Z test_scatter_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5901125Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5902312Z test_send_recv (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5903589Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5904712Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5905377Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5906041Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5906697Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5907289Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5907915Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5908543Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5909144Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5909745Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5910298Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5910920Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5911523Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5912119Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5912771Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5913456Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5914073Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5914555Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5915087Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.5916214Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.5916976Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.5917597Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.5918320Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.5918894Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.5919183Z 2022-11-23T02:13:37.5919275Z Running tests... 2022-11-23T02:13:37.5919818Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.5920525Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2856 2022-11-23T02:13:37.5921241Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2857 2022-11-23T02:13:37.5921840Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.5922616Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.5923246Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.5923951Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.5924458Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.5925000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.5925757Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.5926320Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.5927016Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.5927649Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.5928382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.5929196Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.5930310Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.5931054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.5931752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.5932486Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:13:37.5933624Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:13:37.5934525Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:13:37.5935831Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:13:37.5936814Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:13:37.5937876Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:13:37.5938776Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:13:37.5939908Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:13:37.5940796Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:13:37.5941917Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:13:37.5942814Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:13:37.5944644Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:13:37.5946039Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:13:37.5947817Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:13:37.5949206Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:13:37.5950893Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:13:37.5951977Z ok (5.607s) 2022-11-23T02:13:37.5952341Z 2022-11-23T02:13:37.5952967Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.5953762Z Ran 1 test in 5.608s 2022-11-23T02:13:37.5954165Z 2022-11-23T02:13:37.5954410Z OK 2022-11-23T02:13:37.5954824Z 2022-11-23T02:13:37.5955130Z Generating XML reports... 2022-11-23T02:13:37.5956539Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013825.xml 2022-11-23T02:13:37.5957871Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.5959299Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.5960322Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.5961649Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.5962859Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.5963391Z 2022-11-23T02:13:37.5963671Z Running tests... 2022-11-23T02:13:37.5964757Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.5965767Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.004s) 2022-11-23T02:13:37.5966434Z 2022-11-23T02:13:37.5967068Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.5967956Z Ran 1 test in 0.004s 2022-11-23T02:13:37.5968352Z 2022-11-23T02:13:37.5968626Z OK (skipped=1) 2022-11-23T02:13:37.5968990Z 2022-11-23T02:13:37.5969311Z Generating XML reports... 2022-11-23T02:13:37.5970697Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013835.xml 2022-11-23T02:13:37.5972300Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.5973646Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.5974770Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.5976118Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.5977194Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.5977699Z 2022-11-23T02:13:37.5977991Z Running tests... 2022-11-23T02:13:37.5978986Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.5980186Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3133 2022-11-23T02:13:37.5981376Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3134 2022-11-23T02:13:37.5982413Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.5983965Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.5984984Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.5986336Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.5987388Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.5988406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.5989841Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.5990752Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.5992101Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.5993230Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.5994253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.5995738Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.5997261Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.5998407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.5999492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6000216Z ok (5.409s) 2022-11-23T02:13:37.6000567Z 2022-11-23T02:13:37.6001378Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6002170Z Ran 1 test in 5.409s 2022-11-23T02:13:37.6002636Z 2022-11-23T02:13:37.6002912Z OK 2022-11-23T02:13:37.6003243Z 2022-11-23T02:13:37.6003617Z Generating XML reports... 2022-11-23T02:13:37.6005040Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013839.xml 2022-11-23T02:13:37.6006473Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6007880Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6008918Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6010250Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6011390Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6012071Z 2022-11-23T02:13:37.6012386Z Running tests... 2022-11-23T02:13:37.6013392Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6016016Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.579s) 2022-11-23T02:13:37.6017326Z 2022-11-23T02:13:37.6017954Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6018749Z Ran 1 test in 0.579s 2022-11-23T02:13:37.6019134Z 2022-11-23T02:13:37.6019305Z OK (skipped=1) 2022-11-23T02:13:37.6019667Z 2022-11-23T02:13:37.6019976Z Generating XML reports... 2022-11-23T02:13:37.6021394Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013849.xml 2022-11-23T02:13:37.6022862Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6024269Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6025301Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6026647Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6027596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6028105Z 2022-11-23T02:13:37.6028387Z Running tests... 2022-11-23T02:13:37.6029371Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6030774Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3406 2022-11-23T02:13:37.6032026Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3407 2022-11-23T02:13:37.6033149Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6034607Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6035651Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6036876Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6037931Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6038941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6040425Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6042088Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6043102Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6044417Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6045397Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6046420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6048115Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6049296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6050419Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpodk5fxki 2022-11-23T02:13:37.6051901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpodk5fxki/_remote_module_non_scriptable.py 2022-11-23T02:13:37.6053033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6053744Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2osczx50 2022-11-23T02:13:37.6054299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2osczx50/_remote_module_non_scriptable.py 2022-11-23T02:13:37.6054907Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6055494Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6055946Z ok (5.711s) 2022-11-23T02:13:37.6056151Z 2022-11-23T02:13:37.6056501Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6056925Z Ran 1 test in 5.712s 2022-11-23T02:13:37.6057137Z 2022-11-23T02:13:37.6057285Z OK 2022-11-23T02:13:37.6057399Z 2022-11-23T02:13:37.6057570Z Generating XML reports... 2022-11-23T02:13:37.6058297Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013853.xml 2022-11-23T02:13:37.6059061Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6059794Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6060341Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6061047Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6061677Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6061946Z 2022-11-23T02:13:37.6062036Z Running tests... 2022-11-23T02:13:37.6062564Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6063251Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3679 2022-11-23T02:13:37.6063928Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3680 2022-11-23T02:13:37.6064527Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6065362Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6065920Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6066548Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6067124Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6067674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6068621Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6069397Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6069948Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6070657Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6071221Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6071698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6072525Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6073213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6073820Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpif7q4w2y 2022-11-23T02:13:37.6074433Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpif7q4w2y/_remote_module_non_scriptable.py 2022-11-23T02:13:37.6075029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6075627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3q76il_8 2022-11-23T02:13:37.6076249Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3q76il_8/_remote_module_non_scriptable.py 2022-11-23T02:13:37.6076794Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6077367Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6077830Z ok (5.205s) 2022-11-23T02:13:37.6078021Z 2022-11-23T02:13:37.6078385Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6078811Z Ran 1 test in 5.206s 2022-11-23T02:13:37.6079016Z 2022-11-23T02:13:37.6079157Z OK 2022-11-23T02:13:37.6079337Z 2022-11-23T02:13:37.6079442Z Generating XML reports... 2022-11-23T02:13:37.6080241Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013903.xml 2022-11-23T02:13:37.6080998Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6081738Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6082372Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6083071Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6083637Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6083915Z 2022-11-23T02:13:37.6084071Z Running tests... 2022-11-23T02:13:37.6084539Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6085220Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3952 2022-11-23T02:13:37.6085901Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3953 2022-11-23T02:13:37.6086497Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6087286Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6087889Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6088592Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6089188Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6089745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6090494Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6091035Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6091725Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6092432Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6092981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6093755Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6094588Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6095220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6095779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6096366Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzawid05m 2022-11-23T02:13:37.6097001Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzawid05m/_remote_module_non_scriptable.py 2022-11-23T02:13:37.6097631Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph5exvjhr 2022-11-23T02:13:37.6098264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph5exvjhr/_remote_module_non_scriptable.py 2022-11-23T02:13:37.6098804Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6099381Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6099973Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6100542Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6101109Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6101693Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6102275Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6102845Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6103415Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6103993Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6104588Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6105161Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6105616Z ok (8.312s) 2022-11-23T02:13:37.6105810Z 2022-11-23T02:13:37.6106161Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6106537Z Ran 1 test in 8.312s 2022-11-23T02:13:37.6106746Z 2022-11-23T02:13:37.6106884Z OK 2022-11-23T02:13:37.6107063Z 2022-11-23T02:13:37.6107230Z Generating XML reports... 2022-11-23T02:13:37.6107952Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013912.xml 2022-11-23T02:13:37.6108724Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6109468Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6110103Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6110749Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6111342Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6111611Z 2022-11-23T02:13:37.6111765Z Running tests... 2022-11-23T02:13:37.6112358Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6113029Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4169 2022-11-23T02:13:37.6113723Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4170 2022-11-23T02:13:37.6114322Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6115226Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6115732Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6116463Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6117038Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6117593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6118336Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6118898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6119595Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6120100Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6120657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6121436Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6122234Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6122912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6123480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6124083Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_n9q3ao3 2022-11-23T02:13:37.6124705Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_n9q3ao3/_remote_module_non_scriptable.py 2022-11-23T02:13:37.6125257Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4amhsj8_ 2022-11-23T02:13:37.6125879Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4amhsj8_/_remote_module_non_scriptable.py 2022-11-23T02:13:37.6126531Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6127167Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6127836Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6128429Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6129013Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6129519Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6129970Z ok (5.524s) 2022-11-23T02:13:37.6130251Z 2022-11-23T02:13:37.6130613Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6131053Z Ran 1 test in 5.524s 2022-11-23T02:13:37.6131265Z 2022-11-23T02:13:37.6131403Z OK 2022-11-23T02:13:37.6131581Z 2022-11-23T02:13:37.6131754Z Generating XML reports... 2022-11-23T02:13:37.6132553Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013925.xml 2022-11-23T02:13:37.6133247Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6133984Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6134532Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6135237Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6135889Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6136180Z 2022-11-23T02:13:37.6136412Z Running tests... 2022-11-23T02:13:37.6136949Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6137562Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4384 2022-11-23T02:13:37.6138261Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4385 2022-11-23T02:13:37.6138872Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6139646Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6140206Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6140909Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6141480Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6142023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6142703Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6143340Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6144042Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6144606Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6145155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6145921Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6146729Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6147339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6147838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6148432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr82zt91f 2022-11-23T02:13:37.6149055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr82zt91f/_remote_module_non_scriptable.py 2022-11-23T02:13:37.6149673Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfugprmlk 2022-11-23T02:13:37.6150306Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfugprmlk/_remote_module_non_scriptable.py 2022-11-23T02:13:37.6150909Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6151570Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6152076Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6152661Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6153296Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6153939Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6154516Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6155102Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6155552Z ok (5.909s) 2022-11-23T02:13:37.6155747Z 2022-11-23T02:13:37.6156025Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6156464Z Ran 1 test in 5.910s 2022-11-23T02:13:37.6156734Z 2022-11-23T02:13:37.6156898Z OK 2022-11-23T02:13:37.6157082Z 2022-11-23T02:13:37.6157250Z Generating XML reports... 2022-11-23T02:13:37.6157983Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013935.xml 2022-11-23T02:13:37.6158741Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6159484Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6159968Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6160669Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6161232Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6161500Z 2022-11-23T02:13:37.6161669Z Running tests... 2022-11-23T02:13:37.6162210Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6162977Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4599 2022-11-23T02:13:37.6163692Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4600 2022-11-23T02:13:37.6164298Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6165008Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6165547Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6166237Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6166818Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6167357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6168337Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6169117Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6169614Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6170310Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6170867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6171407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6172187Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6172890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6173528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6174118Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph5mg3g2y 2022-11-23T02:13:37.6174687Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph5mg3g2y/_remote_module_non_scriptable.py 2022-11-23T02:13:37.6175298Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkcqtltub 2022-11-23T02:13:37.6175915Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkcqtltub/_remote_module_non_scriptable.py 2022-11-23T02:13:37.6176518Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6177120Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6177640Z ok (5.727s) 2022-11-23T02:13:37.6177844Z 2022-11-23T02:13:37.6178193Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6178557Z Ran 1 test in 5.727s 2022-11-23T02:13:37.6178771Z 2022-11-23T02:13:37.6178907Z OK 2022-11-23T02:13:37.6179084Z 2022-11-23T02:13:37.6179253Z Generating XML reports... 2022-11-23T02:13:37.6179983Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013945.xml 2022-11-23T02:13:37.6180746Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6181497Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6182053Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6182683Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6183328Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6183608Z 2022-11-23T02:13:37.6183764Z Running tests... 2022-11-23T02:13:37.6184291Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6184990Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4814 2022-11-23T02:13:37.6185779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4815 2022-11-23T02:13:37.6186391Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6187166Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6187653Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6188474Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6189054Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6189593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6190336Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6190882Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6191587Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6192087Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6192627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6193394Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6194356Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6194963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6195528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6196125Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmputjyx26c 2022-11-23T02:13:37.6196746Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmputjyx26c/_remote_module_non_scriptable.py 2022-11-23T02:13:37.6197302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfknndjdm 2022-11-23T02:13:37.6197927Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfknndjdm/_remote_module_non_scriptable.py 2022-11-23T02:13:37.6198603Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6199188Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6199758Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6200336Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6200793Z ok (8.164s) 2022-11-23T02:13:37.6200922Z 2022-11-23T02:13:37.6201272Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6201695Z Ran 1 test in 8.164s 2022-11-23T02:13:37.6201898Z 2022-11-23T02:13:37.6202103Z OK 2022-11-23T02:13:37.6202283Z 2022-11-23T02:13:37.6202460Z Generating XML reports... 2022-11-23T02:13:37.6203263Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013955.xml 2022-11-23T02:13:37.6204028Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6204773Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6205253Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6205971Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6206540Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6206813Z 2022-11-23T02:13:37.6206969Z Running tests... 2022-11-23T02:13:37.6207512Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6208253Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5031 2022-11-23T02:13:37.6208940Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5032 2022-11-23T02:13:37.6209478Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6210253Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6210812Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6211508Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6212073Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6212628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6213369Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6213977Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6214709Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6215284Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6215909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6216677Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6217473Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6218094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6218725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6219311Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp33oxpd4a 2022-11-23T02:13:37.6219955Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp33oxpd4a/_remote_module_non_scriptable.py 2022-11-23T02:13:37.6220583Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4wm8_zdt 2022-11-23T02:13:37.6221201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4wm8_zdt/_remote_module_non_scriptable.py 2022-11-23T02:13:37.6221798Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6222370Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6222954Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6223527Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6223912Z ok (7.508s) 2022-11-23T02:13:37.6224171Z 2022-11-23T02:13:37.6224527Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6224975Z Ran 1 test in 7.508s 2022-11-23T02:13:37.6225183Z 2022-11-23T02:13:37.6225319Z OK 2022-11-23T02:13:37.6225507Z 2022-11-23T02:13:37.6225676Z Generating XML reports... 2022-11-23T02:13:37.6226397Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014007.xml 2022-11-23T02:13:37.6227101Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6227844Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6228403Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6229097Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6229670Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6229940Z 2022-11-23T02:13:37.6230098Z Running tests... 2022-11-23T02:13:37.6230633Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6231264Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5248 2022-11-23T02:13:37.6231983Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5249 2022-11-23T02:13:37.6232579Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6233357Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6233898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6234667Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6235323Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6235863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6236537Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6237179Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6237875Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6238441Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6239063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6239845Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6240706Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6241329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6241835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6242431Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpix6z2mc3 2022-11-23T02:13:37.6243049Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpix6z2mc3/_remote_module_non_scriptable.py 2022-11-23T02:13:37.6243658Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5wlnqk50 2022-11-23T02:13:37.6244268Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5wlnqk50/_remote_module_non_scriptable.py 2022-11-23T02:13:37.6244988Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6245632Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6246230Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6246737Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6247321Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6247940Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6248383Z ok (5.808s) 2022-11-23T02:13:37.6248577Z 2022-11-23T02:13:37.6248974Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6249484Z Ran 1 test in 5.808s 2022-11-23T02:13:37.6249723Z 2022-11-23T02:13:37.6249809Z OK 2022-11-23T02:13:37.6250023Z 2022-11-23T02:13:37.6250219Z Generating XML reports... 2022-11-23T02:13:37.6251072Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014018.xml 2022-11-23T02:13:37.6251981Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6252930Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6253580Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6254422Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6255119Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6255325Z 2022-11-23T02:13:37.6255542Z Running tests... 2022-11-23T02:13:37.6256070Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6257376Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.639s) 2022-11-23T02:13:37.6258156Z 2022-11-23T02:13:37.6258495Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6258924Z Ran 1 test in 0.639s 2022-11-23T02:13:37.6259133Z 2022-11-23T02:13:37.6259287Z OK (skipped=1) 2022-11-23T02:13:37.6259490Z 2022-11-23T02:13:37.6259680Z Generating XML reports... 2022-11-23T02:13:37.6260409Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014028.xml 2022-11-23T02:13:37.6261102Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6261848Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6262479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6263195Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6263761Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6264034Z 2022-11-23T02:13:37.6264193Z Running tests... 2022-11-23T02:13:37.6264734Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6265333Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5529 2022-11-23T02:13:37.6266064Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5530 2022-11-23T02:13:37.6266658Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6267452Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6268001Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6268695Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6269259Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6269811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6270482Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6271103Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6271797Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6272463Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6273011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6273778Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6274575Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6275193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6275694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6276135Z ok (4.908s) 2022-11-23T02:13:37.6276326Z 2022-11-23T02:13:37.6276731Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6277161Z Ran 1 test in 4.908s 2022-11-23T02:13:37.6277365Z 2022-11-23T02:13:37.6277502Z OK 2022-11-23T02:13:37.6277763Z 2022-11-23T02:13:37.6277937Z Generating XML reports... 2022-11-23T02:13:37.6278595Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014033.xml 2022-11-23T02:13:37.6279364Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6280100Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6280650Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6281347Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6281918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6282189Z 2022-11-23T02:13:37.6282339Z Running tests... 2022-11-23T02:13:37.6282927Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6284241Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.579s) 2022-11-23T02:13:37.6285009Z 2022-11-23T02:13:37.6285274Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6285707Z Ran 1 test in 0.579s 2022-11-23T02:13:37.6285910Z 2022-11-23T02:13:37.6286064Z OK (skipped=1) 2022-11-23T02:13:37.6286262Z 2022-11-23T02:13:37.6286501Z Generating XML reports... 2022-11-23T02:13:37.6287226Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014042.xml 2022-11-23T02:13:37.6288128Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6288881Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6289459Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6290375Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6291133Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6291431Z 2022-11-23T02:13:37.6291585Z Running tests... 2022-11-23T02:13:37.6292126Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6292781Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5802 2022-11-23T02:13:37.6293430Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5803 2022-11-23T02:13:37.6294033Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6294748Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6295302Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6295996Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6296560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6297100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6297942Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6298483Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6299116Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6299788Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6300341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6301210Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6302011Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6302622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6303196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6303643Z ok (5.529s) 2022-11-23T02:13:37.6303771Z 2022-11-23T02:13:37.6304187Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6304688Z Ran 1 test in 5.529s 2022-11-23T02:13:37.6304916Z 2022-11-23T02:13:37.6305054Z OK 2022-11-23T02:13:37.6305233Z 2022-11-23T02:13:37.6305402Z Generating XML reports... 2022-11-23T02:13:37.6306130Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014047.xml 2022-11-23T02:13:37.6306902Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6307695Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6308178Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6308869Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6309439Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6309720Z 2022-11-23T02:13:37.6309878Z Running tests... 2022-11-23T02:13:37.6310414Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6310979Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s) 2022-11-23T02:13:37.6311308Z 2022-11-23T02:13:37.6311644Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6312008Z Ran 1 test in 0.002s 2022-11-23T02:13:37.6312213Z 2022-11-23T02:13:37.6312360Z OK (skipped=1) 2022-11-23T02:13:37.6312561Z 2022-11-23T02:13:37.6312735Z Generating XML reports... 2022-11-23T02:13:37.6313456Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014056.xml 2022-11-23T02:13:37.6314225Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6314953Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6315514Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6316147Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6316724Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6316993Z 2022-11-23T02:13:37.6317148Z Running tests... 2022-11-23T02:13:37.6317738Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6318256Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.6318894Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6075 2022-11-23T02:13:37.6319517Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6076 2022-11-23T02:13:37.6320125Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6320989Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6321548Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6322245Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6322808Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6323359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6324105Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6324653Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6325286Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6325919Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6326473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6327251Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6328267Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6328886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6329454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6329975Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjq26753h 2022-11-23T02:13:37.6330595Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjq26753h/_remote_module_non_scriptable.py 2022-11-23T02:13:37.6331229Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzekbzxu2 2022-11-23T02:13:37.6331854Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzekbzxu2/_remote_module_non_scriptable.py 2022-11-23T02:13:37.6332329Z ok (4.929s) 2022-11-23T02:13:37.6332525Z 2022-11-23T02:13:37.6332869Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6333306Z Ran 1 test in 4.929s 2022-11-23T02:13:37.6333512Z 2022-11-23T02:13:37.6333586Z OK 2022-11-23T02:13:37.6333763Z 2022-11-23T02:13:37.6333933Z Generating XML reports... 2022-11-23T02:13:37.6334716Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014100.xml 2022-11-23T02:13:37.6335493Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6336237Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6336790Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6337490Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6338052Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6338333Z 2022-11-23T02:13:37.6338484Z Running tests... 2022-11-23T02:13:37.6339013Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6339546Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.6340200Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6348 2022-11-23T02:13:37.6340829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6349 2022-11-23T02:13:37.6341430Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6342296Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6342790Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6343485Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6344047Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6344584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6345361Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6346147Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6346750Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6347402Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6348047Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6348589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6349348Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6349971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6350574Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi7p0f3n1 2022-11-23T02:13:37.6351193Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi7p0f3n1/_remote_module_non_scriptable.py 2022-11-23T02:13:37.6351789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6352310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps3dp6hq5 2022-11-23T02:13:37.6352938Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps3dp6hq5/_remote_module_non_scriptable.py 2022-11-23T02:13:37.6353537Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6354191Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6354636Z ok (5.205s) 2022-11-23T02:13:37.6354839Z 2022-11-23T02:13:37.6355178Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6355694Z Ran 1 test in 5.205s 2022-11-23T02:13:37.6355834Z 2022-11-23T02:13:37.6355970Z OK 2022-11-23T02:13:37.6356147Z 2022-11-23T02:13:37.6356319Z Generating XML reports... 2022-11-23T02:13:37.6357060Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014109.xml 2022-11-23T02:13:37.6357888Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6358627Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6359183Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6359887Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6360387Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6360663Z 2022-11-23T02:13:37.6360812Z Running tests... 2022-11-23T02:13:37.6361335Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6361895Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.6362580Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6621 2022-11-23T02:13:37.6363311Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6622 2022-11-23T02:13:37.6363907Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6364624Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6365272Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6365970Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6366602Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6367141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6368063Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6368633Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6369345Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6369847Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6370391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6371170Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6371984Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6372592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6373166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6373758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpizfnduvw 2022-11-23T02:13:37.6374324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpizfnduvw/_remote_module_non_scriptable.py 2022-11-23T02:13:37.6374940Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphieduy86 2022-11-23T02:13:37.6375555Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphieduy86/_remote_module_non_scriptable.py 2022-11-23T02:13:37.6376160Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6376742Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.6377187Z ok (5.106s) 2022-11-23T02:13:37.6377381Z 2022-11-23T02:13:37.6377722Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6378149Z Ran 1 test in 5.106s 2022-11-23T02:13:37.6378358Z 2022-11-23T02:13:37.6378510Z OK 2022-11-23T02:13:37.6378688Z 2022-11-23T02:13:37.6378855Z Generating XML reports... 2022-11-23T02:13:37.6379578Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014119.xml 2022-11-23T02:13:37.6380346Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6381095Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6381646Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6382282Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6382847Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6383114Z 2022-11-23T02:13:37.6383279Z Running tests... 2022-11-23T02:13:37.6383976Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6384620Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:13:37.6385264Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6894 2022-11-23T02:13:37.6385896Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6895 2022-11-23T02:13:37.6386494Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6387205Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6387747Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6388518Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6389164Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6389727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6390506Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6391299Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6391783Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6392474Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6393034Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6393585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6394364Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6394972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6395565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf9xyxei7 2022-11-23T02:13:37.6396201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf9xyxei7/_remote_module_non_scriptable.py 2022-11-23T02:13:37.6396731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6397315Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphbvq0frr 2022-11-23T02:13:37.6397938Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphbvq0frr/_remote_module_non_scriptable.py 2022-11-23T02:13:37.6398597Z ok (6.208s) 2022-11-23T02:13:37.6398792Z 2022-11-23T02:13:37.6399130Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6399562Z Ran 1 test in 6.209s 2022-11-23T02:13:37.6399766Z 2022-11-23T02:13:37.6399902Z OK 2022-11-23T02:13:37.6400015Z 2022-11-23T02:13:37.6400200Z Generating XML reports... 2022-11-23T02:13:37.6400924Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014128.xml 2022-11-23T02:13:37.6401685Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6402425Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6402977Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6403672Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6404233Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6404500Z 2022-11-23T02:13:37.6404658Z Running tests... 2022-11-23T02:13:37.6405215Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6405830Z test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7167 2022-11-23T02:13:37.6406450Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7168 2022-11-23T02:13:37.6407050Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6407967Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6408654Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6409288Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6409861Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6410494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6411259Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6411804Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6412507Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6413068Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6413614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6414315Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6415110Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6415734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6416300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6416996Z STAGE:2022-11-23 01:41:41 7168:7168 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6417700Z STAGE:2022-11-23 01:41:41 7167:7167 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6418397Z STAGE:2022-11-23 01:41:41 7168:7168 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6419045Z STAGE:2022-11-23 01:41:41 7168:7168 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6419823Z STAGE:2022-11-23 01:41:41 7167:7167 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6420533Z STAGE:2022-11-23 01:41:41 7167:7167 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6421242Z STAGE:2022-11-23 01:41:41 7168:7168 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6421928Z STAGE:2022-11-23 01:41:41 7167:7167 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6422619Z STAGE:2022-11-23 01:41:41 7168:7168 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6423325Z STAGE:2022-11-23 01:41:41 7167:7167 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6424041Z STAGE:2022-11-23 01:41:41 7168:7168 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6424687Z STAGE:2022-11-23 01:41:41 7167:7167 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6425140Z ok (4.962s) 2022-11-23T02:13:37.6425332Z 2022-11-23T02:13:37.6425664Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6426175Z Ran 1 test in 4.962s 2022-11-23T02:13:37.6426474Z 2022-11-23T02:13:37.6426614Z OK 2022-11-23T02:13:37.6426793Z 2022-11-23T02:13:37.6426960Z Generating XML reports... 2022-11-23T02:13:37.6427766Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014138.xml 2022-11-23T02:13:37.6428464Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6429202Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6429417Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6429925Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6430160Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6430166Z 2022-11-23T02:13:37.6430334Z Running tests... 2022-11-23T02:13:37.6430744Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6431136Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7380 2022-11-23T02:13:37.6431399Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7381 2022-11-23T02:13:37.6431711Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6432152Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6432370Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6432822Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6433057Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6433348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6433781Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6434002Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6434448Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6434681Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6434894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6435352Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6435817Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6436094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6436482Z STAGE:2022-11-23 01:41:50 7381:7381 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6436750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6437136Z STAGE:2022-11-23 01:41:50 7380:7380 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6437530Z STAGE:2022-11-23 01:41:50 7381:7381 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6437946Z STAGE:2022-11-23 01:41:50 7381:7381 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6438350Z STAGE:2022-11-23 01:41:50 7380:7380 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6438756Z STAGE:2022-11-23 01:41:50 7380:7380 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6439227Z STAGE:2022-11-23 01:41:50 7381:7381 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6439617Z STAGE:2022-11-23 01:41:50 7380:7380 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6440011Z STAGE:2022-11-23 01:41:50 7381:7381 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6440424Z STAGE:2022-11-23 01:41:50 7381:7381 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6440960Z STAGE:2022-11-23 01:41:50 7380:7380 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6441380Z STAGE:2022-11-23 01:41:50 7380:7380 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6441771Z STAGE:2022-11-23 01:41:50 7381:7381 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6442156Z STAGE:2022-11-23 01:41:50 7380:7380 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6442615Z STAGE:2022-11-23 01:41:50 7381:7381 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6443042Z STAGE:2022-11-23 01:41:50 7381:7381 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6443436Z STAGE:2022-11-23 01:41:50 7380:7380 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6443840Z STAGE:2022-11-23 01:41:50 7380:7380 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6444675Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:13:37.6444837Z warnings.warn( 2022-11-23T02:13:37.6445647Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:13:37.6445744Z warnings.warn( 2022-11-23T02:13:37.6445896Z ok (5.008s) 2022-11-23T02:13:37.6445902Z 2022-11-23T02:13:37.6446237Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6446395Z Ran 1 test in 5.008s 2022-11-23T02:13:37.6446402Z 2022-11-23T02:13:37.6446546Z OK 2022-11-23T02:13:37.6446552Z 2022-11-23T02:13:37.6446722Z Generating XML reports... 2022-11-23T02:13:37.6447249Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014148.xml 2022-11-23T02:13:37.6447629Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6448128Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6448357Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6448808Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6449042Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6449049Z 2022-11-23T02:13:37.6449211Z Running tests... 2022-11-23T02:13:37.6449556Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6449929Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7593 2022-11-23T02:13:37.6450186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7594 2022-11-23T02:13:37.6450493Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6450932Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6451238Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6451689Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6451935Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6452274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6452667Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6453101Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6453321Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6453864Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6454112Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6454395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6454880Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6455149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6455417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6455697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:13:37.6455972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:13:37.6456435Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.6456974Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.6457362Z STAGE:2022-11-23 01:41:59 7594:7594 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6457754Z STAGE:2022-11-23 01:41:59 7593:7593 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6458147Z STAGE:2022-11-23 01:41:59 7594:7594 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6458557Z STAGE:2022-11-23 01:41:59 7594:7594 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6458953Z STAGE:2022-11-23 01:41:59 7593:7593 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6459360Z STAGE:2022-11-23 01:41:59 7593:7593 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6459759Z STAGE:2022-11-23 01:41:59 7594:7594 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6460145Z STAGE:2022-11-23 01:41:59 7593:7593 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6460550Z STAGE:2022-11-23 01:41:59 7594:7594 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6460962Z STAGE:2022-11-23 01:41:59 7594:7594 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6461359Z STAGE:2022-11-23 01:41:59 7593:7593 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6461700Z STAGE:2022-11-23 01:41:59 7593:7593 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6462081Z STAGE:2022-11-23 01:41:59 7594:7594 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6462465Z STAGE:2022-11-23 01:41:59 7593:7593 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6462944Z STAGE:2022-11-23 01:41:59 7594:7594 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6463568Z STAGE:2022-11-23 01:41:59 7594:7594 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:41:59 7593:7593 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6463577Z 2022-11-23T02:13:37.6463991Z STAGE:2022-11-23 01:41:59 7593:7593 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6464806Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:13:37.6465023Z warnings.warn( 2022-11-23T02:13:37.6465894Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:13:37.6466131Z warnings.warn( 2022-11-23T02:13:37.6466279Z ok (4.919s) 2022-11-23T02:13:37.6466286Z 2022-11-23T02:13:37.6466625Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6466790Z Ran 1 test in 4.919s 2022-11-23T02:13:37.6466797Z 2022-11-23T02:13:37.6466934Z OK 2022-11-23T02:13:37.6466940Z 2022-11-23T02:13:37.6467108Z Generating XML reports... 2022-11-23T02:13:37.6467619Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014157.xml 2022-11-23T02:13:37.6467999Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6468436Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6468666Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6469112Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6469354Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6469361Z 2022-11-23T02:13:37.6469517Z Running tests... 2022-11-23T02:13:37.6469851Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6470216Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7812 2022-11-23T02:13:37.6470477Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7813 2022-11-23T02:13:37.6470782Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6471229Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6471388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6471843Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6472080Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6472355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6472789Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6473008Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6473453Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6473681Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6474049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6474517Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6474973Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6475242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6475691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6476085Z skip: Skipped due to small world size. (4.926s) 2022-11-23T02:13:37.6476092Z 2022-11-23T02:13:37.6476536Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6476691Z Ran 1 test in 4.927s 2022-11-23T02:13:37.6476698Z 2022-11-23T02:13:37.6476867Z OK (skipped=1) 2022-11-23T02:13:37.6476878Z 2022-11-23T02:13:37.6477107Z Generating XML reports... 2022-11-23T02:13:37.6477641Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014206.xml 2022-11-23T02:13:37.6478019Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6478451Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6478672Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6479051Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6479286Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6479292Z 2022-11-23T02:13:37.6479462Z Running tests... 2022-11-23T02:13:37.6479792Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6480176Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8019 2022-11-23T02:13:37.6480435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8020 2022-11-23T02:13:37.6480737Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6481173Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6481393Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6481924Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6482156Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6482432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6482901Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6483331Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6483549Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6483998Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6484230Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6484519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6484980Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6485250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6485718Z STAGE:2022-11-23 01:42:17 8020:8020 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6486056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6486445Z STAGE:2022-11-23 01:42:18 8019:8019 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6486770Z STAGE:2022-11-23 01:42:18 8019:8019 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6487186Z STAGE:2022-11-23 01:42:18 8019:8019 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6487585Z STAGE:2022-11-23 01:42:18 8020:8020 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6488044Z STAGE:2022-11-23 01:42:18 8020:8020 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6488503Z STAGE:2022-11-23 01:42:18 8019:8019 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6488914Z STAGE:2022-11-23 01:42:18 8020:8020 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6489309Z STAGE:2022-11-23 01:42:18 8019:8019 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6489720Z STAGE:2022-11-23 01:42:18 8019:8019 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6490108Z STAGE:2022-11-23 01:42:18 8020:8020 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6490528Z STAGE:2022-11-23 01:42:18 8020:8020 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6490917Z STAGE:2022-11-23 01:42:18 8019:8019 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6491300Z STAGE:2022-11-23 01:42:18 8020:8020 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6491697Z STAGE:2022-11-23 01:42:18 8019:8019 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6492112Z STAGE:2022-11-23 01:42:18 8019:8019 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6492503Z STAGE:2022-11-23 01:42:18 8020:8020 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6492911Z STAGE:2022-11-23 01:42:18 8020:8020 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6493739Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:13:37.6493894Z warnings.warn( 2022-11-23T02:13:37.6494702Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:13:37.6494859Z warnings.warn( 2022-11-23T02:13:37.6495007Z ok (5.105s) 2022-11-23T02:13:37.6495014Z 2022-11-23T02:13:37.6495354Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6495510Z Ran 1 test in 5.106s 2022-11-23T02:13:37.6495516Z 2022-11-23T02:13:37.6495656Z OK 2022-11-23T02:13:37.6495662Z 2022-11-23T02:13:37.6495847Z Generating XML reports... 2022-11-23T02:13:37.6496287Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014215.xml 2022-11-23T02:13:37.6496748Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6497187Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6497409Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6497942Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6498178Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6498184Z 2022-11-23T02:13:37.6498405Z Running tests... 2022-11-23T02:13:37.6498750Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6499124Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8232 2022-11-23T02:13:37.6499383Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8233 2022-11-23T02:13:37.6499699Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6500218Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6500503Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6500967Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6501213Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6501491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6501931Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6502150Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6502595Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6502828Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6503112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6503586Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6504042Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6504246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6504636Z STAGE:2022-11-23 01:42:27 8233:8233 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6504910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6505303Z STAGE:2022-11-23 01:42:27 8232:8232 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6505700Z STAGE:2022-11-23 01:42:27 8232:8232 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6506099Z STAGE:2022-11-23 01:42:27 8233:8233 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6506510Z STAGE:2022-11-23 01:42:27 8232:8232 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6506926Z STAGE:2022-11-23 01:42:27 8233:8233 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6507735Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:13:37.6507892Z warnings.warn( 2022-11-23T02:13:37.6508689Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:13:37.6508919Z warnings.warn( 2022-11-23T02:13:37.6509069Z ok (4.943s) 2022-11-23T02:13:37.6509075Z 2022-11-23T02:13:37.6509414Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6509644Z Ran 1 test in 4.943s 2022-11-23T02:13:37.6509651Z 2022-11-23T02:13:37.6509789Z OK 2022-11-23T02:13:37.6509795Z 2022-11-23T02:13:37.6509965Z Generating XML reports... 2022-11-23T02:13:37.6510473Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014224.xml 2022-11-23T02:13:37.6510847Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6511283Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6511497Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6512009Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6512259Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6512266Z 2022-11-23T02:13:37.6512417Z Running tests... 2022-11-23T02:13:37.6512685Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6513050Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8445 2022-11-23T02:13:37.6513310Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8446 2022-11-23T02:13:37.6513616Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6514054Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6514286Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6514736Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6514966Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6515244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6515675Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6515896Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6516342Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6516580Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6516866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6517331Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6517857Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6518135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6518402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6518793Z STAGE:2022-11-23 01:42:36 8445:8445 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6519175Z STAGE:2022-11-23 01:42:36 8446:8446 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6519578Z STAGE:2022-11-23 01:42:36 8445:8445 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6519988Z STAGE:2022-11-23 01:42:36 8445:8445 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6520522Z STAGE:2022-11-23 01:42:36 8446:8446 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6520863Z STAGE:2022-11-23 01:42:36 8446:8446 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6521246Z STAGE:2022-11-23 01:42:36 8445:8445 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6521630Z STAGE:2022-11-23 01:42:36 8446:8446 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6522021Z STAGE:2022-11-23 01:42:36 8445:8445 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6522413Z STAGE:2022-11-23 01:42:36 8446:8446 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6522829Z STAGE:2022-11-23 01:42:36 8445:8445 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6523234Z STAGE:2022-11-23 01:42:36 8446:8446 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6523443Z ok (4.810s) 2022-11-23T02:13:37.6523450Z 2022-11-23T02:13:37.6523880Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6524038Z Ran 1 test in 4.810s 2022-11-23T02:13:37.6524045Z 2022-11-23T02:13:37.6524183Z OK 2022-11-23T02:13:37.6524189Z 2022-11-23T02:13:37.6524360Z Generating XML reports... 2022-11-23T02:13:37.6524880Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014233.xml 2022-11-23T02:13:37.6525257Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6525691Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6525910Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6526351Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6526595Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6526601Z 2022-11-23T02:13:37.6526752Z Running tests... 2022-11-23T02:13:37.6527084Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6527402Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s) 2022-11-23T02:13:37.6527408Z 2022-11-23T02:13:37.6527937Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6528031Z Ran 1 test in 0.002s 2022-11-23T02:13:37.6528110Z 2022-11-23T02:13:37.6528198Z OK (skipped=1) 2022-11-23T02:13:37.6528203Z 2022-11-23T02:13:37.6528377Z Generating XML reports... 2022-11-23T02:13:37.6529076Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014242.xml 2022-11-23T02:13:37.6529555Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6530076Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6530406Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6530999Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6531295Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6531330Z 2022-11-23T02:13:37.6531523Z Running tests... 2022-11-23T02:13:37.6532044Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6532431Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s) 2022-11-23T02:13:37.6532439Z 2022-11-23T02:13:37.6532839Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6533024Z Ran 1 test in 0.002s 2022-11-23T02:13:37.6533162Z 2022-11-23T02:13:37.6533353Z OK (skipped=1) 2022-11-23T02:13:37.6533360Z 2022-11-23T02:13:37.6533567Z Generating XML reports... 2022-11-23T02:13:37.6534187Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014246.xml 2022-11-23T02:13:37.6534638Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6535156Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6535417Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6535948Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6536240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6536248Z 2022-11-23T02:13:37.6536431Z Running tests... 2022-11-23T02:13:37.6536825Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6537277Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8790 2022-11-23T02:13:37.6537592Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8791 2022-11-23T02:13:37.6537964Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6538498Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6538766Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6539319Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6539602Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6540032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6540558Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6540820Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6541352Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6541627Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6541977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6542531Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6543070Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6543395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6543714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6544112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:13:37.6544451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:13:37.6544969Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.6545436Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.6546030Z STAGE:2022-11-23 01:42:53 8790:8790 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:42:53 8791:8791 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6546104Z 2022-11-23T02:13:37.6546515Z STAGE:2022-11-23 01:42:53 8791:8791 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6546911Z STAGE:2022-11-23 01:42:53 8790:8790 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6547319Z STAGE:2022-11-23 01:42:53 8791:8791 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6547654Z STAGE:2022-11-23 01:42:53 8790:8790 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6548038Z STAGE:2022-11-23 01:42:53 8791:8791 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6548423Z STAGE:2022-11-23 01:42:53 8790:8790 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6548827Z STAGE:2022-11-23 01:42:53 8790:8790 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6549288Z STAGE:2022-11-23 01:42:53 8790:8790 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6549700Z STAGE:2022-11-23 01:42:53 8791:8791 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6550113Z STAGE:2022-11-23 01:42:53 8791:8791 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6550265Z ok (5.017s) 2022-11-23T02:13:37.6550272Z 2022-11-23T02:13:37.6550608Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6550762Z Ran 1 test in 5.017s 2022-11-23T02:13:37.6550769Z 2022-11-23T02:13:37.6550921Z OK 2022-11-23T02:13:37.6550927Z 2022-11-23T02:13:37.6551097Z Generating XML reports... 2022-11-23T02:13:37.6551606Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014250.xml 2022-11-23T02:13:37.6551982Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6552418Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6552639Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6553085Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6553331Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6553338Z 2022-11-23T02:13:37.6553494Z Running tests... 2022-11-23T02:13:37.6553825Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6554272Z test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9009 2022-11-23T02:13:37.6554529Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9010 2022-11-23T02:13:37.6554774Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6555302Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6555566Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6556162Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6556448Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6556777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6557331Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6557847Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6558103Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6558740Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6559019Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6559375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6559931Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6560250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6560572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6560813Z skip: Skipped due to small world size. (5.316s) 2022-11-23T02:13:37.6560821Z 2022-11-23T02:13:37.6561217Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6561403Z Ran 1 test in 5.316s 2022-11-23T02:13:37.6561417Z 2022-11-23T02:13:37.6561737Z OK (skipped=1) 2022-11-23T02:13:37.6561747Z 2022-11-23T02:13:37.6561961Z Generating XML reports... 2022-11-23T02:13:37.6562588Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014259.xml 2022-11-23T02:13:37.6563047Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6563565Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6563750Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6564282Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6564571Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6564579Z 2022-11-23T02:13:37.6564775Z Running tests... 2022-11-23T02:13:37.6565119Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6565458Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s) 2022-11-23T02:13:37.6565466Z 2022-11-23T02:13:37.6565793Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6565951Z Ran 1 test in 0.002s 2022-11-23T02:13:37.6565957Z 2022-11-23T02:13:37.6566110Z OK (skipped=1) 2022-11-23T02:13:37.6566116Z 2022-11-23T02:13:37.6566284Z Generating XML reports... 2022-11-23T02:13:37.6566792Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014309.xml 2022-11-23T02:13:37.6567243Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6567674Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6567960Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6568410Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6568751Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6568757Z 2022-11-23T02:13:37.6568911Z Running tests... 2022-11-23T02:13:37.6569245Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6569595Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s) 2022-11-23T02:13:37.6569602Z 2022-11-23T02:13:37.6569927Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6570083Z Ran 1 test in 0.002s 2022-11-23T02:13:37.6570089Z 2022-11-23T02:13:37.6570240Z OK (skipped=1) 2022-11-23T02:13:37.6570246Z 2022-11-23T02:13:37.6570419Z Generating XML reports... 2022-11-23T02:13:37.6570942Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014313.xml 2022-11-23T02:13:37.6571327Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6571764Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6571998Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6572445Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6572679Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6572687Z 2022-11-23T02:13:37.6572840Z Running tests... 2022-11-23T02:13:37.6573171Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6573555Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s) 2022-11-23T02:13:37.6573571Z 2022-11-23T02:13:37.6573915Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6574083Z Ran 1 test in 0.002s 2022-11-23T02:13:37.6574089Z 2022-11-23T02:13:37.6574245Z OK (skipped=1) 2022-11-23T02:13:37.6574250Z 2022-11-23T02:13:37.6574426Z Generating XML reports... 2022-11-23T02:13:37.6574927Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014317.xml 2022-11-23T02:13:37.6575303Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6575739Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6575962Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6576409Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6576657Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6576663Z 2022-11-23T02:13:37.6576819Z Running tests... 2022-11-23T02:13:37.6577219Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6577554Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s) 2022-11-23T02:13:37.6577562Z 2022-11-23T02:13:37.6577890Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6577981Z Ran 1 test in 0.002s 2022-11-23T02:13:37.6578050Z 2022-11-23T02:13:37.6578140Z OK (skipped=1) 2022-11-23T02:13:37.6578146Z 2022-11-23T02:13:37.6578316Z Generating XML reports... 2022-11-23T02:13:37.6578833Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014321.xml 2022-11-23T02:13:37.6579213Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6579653Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6579874Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6580317Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6580551Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6580557Z 2022-11-23T02:13:37.6580709Z Running tests... 2022-11-23T02:13:37.6581036Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6581419Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9480 2022-11-23T02:13:37.6581679Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9481 2022-11-23T02:13:37.6582065Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6582507Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6582724Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6583165Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6583395Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6583688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6584151Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6584635Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6584868Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6585315Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6585482Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6585756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6586212Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6586497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6586843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6586991Z ok (4.910s) 2022-11-23T02:13:37.6586998Z 2022-11-23T02:13:37.6587403Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6587563Z Ran 1 test in 4.911s 2022-11-23T02:13:37.6587569Z 2022-11-23T02:13:37.6587706Z OK 2022-11-23T02:13:37.6587712Z 2022-11-23T02:13:37.6587883Z Generating XML reports... 2022-11-23T02:13:37.6588401Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014325.xml 2022-11-23T02:13:37.6588776Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6589208Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6589431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6589870Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6590105Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6590118Z 2022-11-23T02:13:37.6590272Z Running tests... 2022-11-23T02:13:37.6590604Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6590981Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9687 2022-11-23T02:13:37.6591239Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9688 2022-11-23T02:13:37.6591545Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6591980Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6592137Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6592577Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6592889Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6593167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6593646Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6594161Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6594382Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6594831Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6595064Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6595347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6595860Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6596158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6596431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6596705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:13:37.6596980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:13:37.6597443Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.6597960Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.6598240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T02:13:37.6598535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T02:13:37.6598993Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T02:13:37.6599452Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T02:13:37.6599730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T02:13:37.6600185Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T02:13:37.6600461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T02:13:37.6600922Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T02:13:37.6601012Z ok (5.058s) 2022-11-23T02:13:37.6601022Z 2022-11-23T02:13:37.6601352Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6601521Z Ran 1 test in 5.059s 2022-11-23T02:13:37.6601527Z 2022-11-23T02:13:37.6601665Z OK 2022-11-23T02:13:37.6601671Z 2022-11-23T02:13:37.6601840Z Generating XML reports... 2022-11-23T02:13:37.6602339Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014334.xml 2022-11-23T02:13:37.6602712Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6603144Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6603367Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6603822Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6604131Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6604138Z 2022-11-23T02:13:37.6604295Z Running tests... 2022-11-23T02:13:37.6604639Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6604936Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports all_gather_v (0.002s) 2022-11-23T02:13:37.6604943Z 2022-11-23T02:13:37.6605273Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6605427Z Ran 1 test in 0.003s 2022-11-23T02:13:37.6605433Z 2022-11-23T02:13:37.6605603Z OK (skipped=1) 2022-11-23T02:13:37.6605609Z 2022-11-23T02:13:37.6605781Z Generating XML reports... 2022-11-23T02:13:37.6606355Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014343.xml 2022-11-23T02:13:37.6606728Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6607227Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6607459Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6607975Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6608294Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6608301Z 2022-11-23T02:13:37.6608478Z Running tests... 2022-11-23T02:13:37.6608815Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6609192Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9984 2022-11-23T02:13:37.6609450Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9985 2022-11-23T02:13:37.6609760Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6610210Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6610433Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6610877Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6611117Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6611493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6611930Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6612153Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6612600Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6612839Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6613114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6613583Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6614035Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6614307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6614576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6614857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:13:37.6615141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:13:37.6615624Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.6616089Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.6616490Z STAGE:2022-11-23 01:43:50 9985:9985 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6616873Z STAGE:2022-11-23 01:43:50 9984:9984 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6617265Z STAGE:2022-11-23 01:43:50 9984:9984 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6617664Z STAGE:2022-11-23 01:43:50 9985:9985 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6618075Z STAGE:2022-11-23 01:43:50 9984:9984 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6618546Z STAGE:2022-11-23 01:43:50 9985:9985 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6618946Z STAGE:2022-11-23 01:43:50 9985:9985 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6619341Z STAGE:2022-11-23 01:43:50 9984:9984 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6619804Z STAGE:2022-11-23 01:43:50 9985:9985 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6620206Z STAGE:2022-11-23 01:43:50 9985:9985 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6620597Z STAGE:2022-11-23 01:43:50 9984:9984 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6621006Z STAGE:2022-11-23 01:43:50 9984:9984 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6621818Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:13:37.6621979Z warnings.warn( 2022-11-23T02:13:37.6622782Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:13:37.6622952Z warnings.warn( 2022-11-23T02:13:37.6623099Z ok (5.015s) 2022-11-23T02:13:37.6623105Z 2022-11-23T02:13:37.6623438Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6623596Z Ran 1 test in 5.015s 2022-11-23T02:13:37.6623602Z 2022-11-23T02:13:37.6623742Z OK 2022-11-23T02:13:37.6623748Z 2022-11-23T02:13:37.6623918Z Generating XML reports... 2022-11-23T02:13:37.6624428Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014347.xml 2022-11-23T02:13:37.6624817Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6625183Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6625405Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6625845Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6626080Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6626086Z 2022-11-23T02:13:37.6626240Z Running tests... 2022-11-23T02:13:37.6626573Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6626958Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10203 2022-11-23T02:13:37.6627301Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10204 2022-11-23T02:13:37.6627606Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6628048Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6628266Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6628711Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6628944Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6629223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6629725Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6629960Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6630473Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6630709Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6630984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6631447Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6631902Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6632173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6632458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6632673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:13:37.6633024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:13:37.6633487Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.6633939Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.6634421Z STAGE:2022-11-23 01:43:59 10203:10203 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6634811Z STAGE:2022-11-23 01:43:59 10204:10204 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6635209Z STAGE:2022-11-23 01:43:59 10203:10203 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6635865Z STAGE:2022-11-23 01:43:59 10204:10204 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:43:59 10203:10203 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6635873Z 2022-11-23T02:13:37.6636289Z STAGE:2022-11-23 01:43:59 10204:10204 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6636682Z STAGE:2022-11-23 01:43:59 10203:10203 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6637070Z STAGE:2022-11-23 01:43:59 10204:10204 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6637469Z STAGE:2022-11-23 01:43:59 10204:10204 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6637884Z STAGE:2022-11-23 01:43:59 10204:10204 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6638279Z STAGE:2022-11-23 01:43:59 10203:10203 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6638781Z STAGE:2022-11-23 01:43:59 10203:10203 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6639591Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:13:37.6639750Z warnings.warn( 2022-11-23T02:13:37.6640555Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:13:37.6640709Z warnings.warn( 2022-11-23T02:13:37.6640858Z ok (5.007s) 2022-11-23T02:13:37.6640865Z 2022-11-23T02:13:37.6641196Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6641426Z Ran 1 test in 5.008s 2022-11-23T02:13:37.6641435Z 2022-11-23T02:13:37.6641577Z OK 2022-11-23T02:13:37.6641583Z 2022-11-23T02:13:37.6641758Z Generating XML reports... 2022-11-23T02:13:37.6642270Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014356.xml 2022-11-23T02:13:37.6642711Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6643150Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6643373Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6643751Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6643982Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6643988Z 2022-11-23T02:13:37.6644158Z Running tests... 2022-11-23T02:13:37.6644492Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6644877Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10422 2022-11-23T02:13:37.6645142Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10423 2022-11-23T02:13:37.6645455Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6645886Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6646110Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6646572Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6646811Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6647100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6647558Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6648045Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6648314Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6648767Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6649000Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6649288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6649749Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6650104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6650378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6650656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:13:37.6650931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:13:37.6651324Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.6651780Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.6652233Z STAGE:2022-11-23 01:44:08 10422:10422 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6653292Z STAGE:2022-11-23 01:44:08 10423:10423 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6653713Z STAGE:2022-11-23 01:44:08 10422:10422 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6654124Z STAGE:2022-11-23 01:44:08 10422:10422 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6654579Z STAGE:2022-11-23 01:44:08 10423:10423 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6654987Z STAGE:2022-11-23 01:44:08 10423:10423 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6655374Z STAGE:2022-11-23 01:44:08 10422:10422 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6655771Z STAGE:2022-11-23 01:44:08 10423:10423 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6656170Z STAGE:2022-11-23 01:44:08 10422:10422 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6656587Z STAGE:2022-11-23 01:44:08 10422:10422 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6656984Z STAGE:2022-11-23 01:44:08 10423:10423 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6657392Z STAGE:2022-11-23 01:44:08 10423:10423 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6658205Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:13:37.6658361Z warnings.warn( 2022-11-23T02:13:37.6659188Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:13:37.6659349Z warnings.warn( 2022-11-23T02:13:37.6659491Z ok (5.130s) 2022-11-23T02:13:37.6659498Z 2022-11-23T02:13:37.6659830Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6659986Z Ran 1 test in 5.131s 2022-11-23T02:13:37.6659992Z 2022-11-23T02:13:37.6660126Z OK 2022-11-23T02:13:37.6660133Z 2022-11-23T02:13:37.6660307Z Generating XML reports... 2022-11-23T02:13:37.6660964Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014405.xml 2022-11-23T02:13:37.6661352Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6661782Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6661940Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6662458Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6662709Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6662716Z 2022-11-23T02:13:37.6662867Z Running tests... 2022-11-23T02:13:37.6663202Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6663578Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10641 2022-11-23T02:13:37.6663852Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10642 2022-11-23T02:13:37.6664155Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6664590Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6664868Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6665326Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6665564Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6665909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6666357Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6666573Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6667019Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6667251Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6667534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6668003Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6668451Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6668730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6668999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6669278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:13:37.6669489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:13:37.6669940Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.6670397Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.6670789Z STAGE:2022-11-23 01:44:17 10641:10641 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6671178Z STAGE:2022-11-23 01:44:17 10642:10642 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6671577Z STAGE:2022-11-23 01:44:17 10642:10642 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6672000Z STAGE:2022-11-23 01:44:17 10642:10642 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6672395Z STAGE:2022-11-23 01:44:17 10641:10641 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6672851Z STAGE:2022-11-23 01:44:17 10641:10641 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6673237Z STAGE:2022-11-23 01:44:17 10642:10642 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6673704Z STAGE:2022-11-23 01:44:17 10641:10641 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6674104Z STAGE:2022-11-23 01:44:17 10642:10642 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6674512Z STAGE:2022-11-23 01:44:17 10642:10642 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6674917Z STAGE:2022-11-23 01:44:17 10641:10641 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6675326Z STAGE:2022-11-23 01:44:17 10641:10641 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6676132Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:13:37.6676296Z warnings.warn( 2022-11-23T02:13:37.6677153Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:13:37.6677322Z warnings.warn( 2022-11-23T02:13:37.6677470Z ok (5.560s) 2022-11-23T02:13:37.6677476Z 2022-11-23T02:13:37.6677887Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6678045Z Ran 1 test in 5.561s 2022-11-23T02:13:37.6678051Z 2022-11-23T02:13:37.6678192Z OK 2022-11-23T02:13:37.6678198Z 2022-11-23T02:13:37.6678369Z Generating XML reports... 2022-11-23T02:13:37.6678873Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014414.xml 2022-11-23T02:13:37.6679247Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6679621Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6679845Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6680296Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6680531Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6680538Z 2022-11-23T02:13:37.6680690Z Running tests... 2022-11-23T02:13:37.6681018Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6681393Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10860 2022-11-23T02:13:37.6681649Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10861 2022-11-23T02:13:37.6681955Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6682394Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6682622Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6683063Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6683297Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6683577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6684113Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6684546Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6684876Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6685339Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6685573Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6685851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6686312Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6686583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6686850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6686990Z skip: Skipped due to small world size. (4.808s) 2022-11-23T02:13:37.6687059Z 2022-11-23T02:13:37.6687327Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6687557Z Ran 1 test in 4.808s 2022-11-23T02:13:37.6687565Z 2022-11-23T02:13:37.6687777Z OK (skipped=1) 2022-11-23T02:13:37.6687784Z 2022-11-23T02:13:37.6688026Z Generating XML reports... 2022-11-23T02:13:37.6688547Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014424.xml 2022-11-23T02:13:37.6688923Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6689353Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6689573Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6690026Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6690258Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6690269Z 2022-11-23T02:13:37.6690424Z Running tests... 2022-11-23T02:13:37.6690758Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6691128Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11067 2022-11-23T02:13:37.6691387Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11068 2022-11-23T02:13:37.6691689Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6692124Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6692350Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6692794Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6693034Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6693316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6693745Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6693963Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6694336Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6694573Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6694925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6695390Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6695844Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6696198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6696468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6696674Z skip: Skipped due to small world size. (4.825s) 2022-11-23T02:13:37.6696681Z 2022-11-23T02:13:37.6697019Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6697184Z Ran 1 test in 4.825s 2022-11-23T02:13:37.6697190Z 2022-11-23T02:13:37.6697342Z OK (skipped=1) 2022-11-23T02:13:37.6697348Z 2022-11-23T02:13:37.6697516Z Generating XML reports... 2022-11-23T02:13:37.6698023Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014433.xml 2022-11-23T02:13:37.6698460Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6698964Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6699194Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6699656Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6699887Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6699894Z 2022-11-23T02:13:37.6700045Z Running tests... 2022-11-23T02:13:37.6700378Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6700762Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11274 2022-11-23T02:13:37.6701020Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11275 2022-11-23T02:13:37.6701329Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6701702Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6701929Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6702378Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6702607Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6702887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6703348Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6703777Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6703999Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6704444Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6704685Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6704962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6705493Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6705764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6706031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6706233Z skip: Skipped due to small world size. (5.218s) 2022-11-23T02:13:37.6706241Z 2022-11-23T02:13:37.6706571Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6706814Z Ran 1 test in 5.218s 2022-11-23T02:13:37.6706821Z 2022-11-23T02:13:37.6706975Z OK (skipped=1) 2022-11-23T02:13:37.6706982Z 2022-11-23T02:13:37.6707150Z Generating XML reports... 2022-11-23T02:13:37.6707664Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014442.xml 2022-11-23T02:13:37.6708036Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6708534Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6708690Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6709131Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6709363Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6709375Z 2022-11-23T02:13:37.6709605Z Running tests... 2022-11-23T02:13:37.6709954Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6710328Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11481 2022-11-23T02:13:37.6710587Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11482 2022-11-23T02:13:37.6710895Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6711330Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6711545Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6711996Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6712236Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6712523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6712954Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6713170Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6713608Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6713839Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6714125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6714588Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6715048Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6715320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6715587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6715841Z skip: Skipped due to small world size. (5.112s) 2022-11-23T02:13:37.6715848Z 2022-11-23T02:13:37.6716181Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6716274Z Ran 1 test in 5.112s 2022-11-23T02:13:37.6716280Z 2022-11-23T02:13:37.6716432Z OK (skipped=1) 2022-11-23T02:13:37.6716438Z 2022-11-23T02:13:37.6716619Z Generating XML reports... 2022-11-23T02:13:37.6717125Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014451.xml 2022-11-23T02:13:37.6717502Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6718016Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6718238Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6718684Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6718978Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6718985Z 2022-11-23T02:13:37.6719148Z Running tests... 2022-11-23T02:13:37.6719477Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6719839Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11688 2022-11-23T02:13:37.6720101Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11689 2022-11-23T02:13:37.6720461Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6720920Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6721138Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6721590Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6721823Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6722100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6722562Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6722991Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6723220Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6723596Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6723833Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6724126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6724586Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6724853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6725246Z STAGE:2022-11-23 01:45:03 11689:11689 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6725515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6725911Z STAGE:2022-11-23 01:45:04 11688:11688 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6726316Z STAGE:2022-11-23 01:45:04 11688:11688 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6726791Z STAGE:2022-11-23 01:45:04 11688:11688 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6727201Z STAGE:2022-11-23 01:45:04 11689:11689 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6727611Z STAGE:2022-11-23 01:45:04 11689:11689 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6728141Z STAGE:2022-11-23 01:45:04 11688:11688 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6728537Z STAGE:2022-11-23 01:45:04 11689:11689 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6728931Z STAGE:2022-11-23 01:45:04 11688:11688 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6729345Z STAGE:2022-11-23 01:45:04 11688:11688 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6729834Z STAGE:2022-11-23 01:45:04 11689:11689 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6730321Z STAGE:2022-11-23 01:45:04 11689:11689 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6731131Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:13:37.6731289Z warnings.warn( 2022-11-23T02:13:37.6732080Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:13:37.6732301Z warnings.warn( 2022-11-23T02:13:37.6732496Z ok (5.061s) 2022-11-23T02:13:37.6732504Z 2022-11-23T02:13:37.6732889Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6733049Z Ran 1 test in 5.062s 2022-11-23T02:13:37.6733055Z 2022-11-23T02:13:37.6733129Z OK 2022-11-23T02:13:37.6733200Z 2022-11-23T02:13:37.6733305Z Generating XML reports... 2022-11-23T02:13:37.6733815Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014501.xml 2022-11-23T02:13:37.6734190Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6734625Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6734848Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6735309Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6735551Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6735557Z 2022-11-23T02:13:37.6735714Z Running tests... 2022-11-23T02:13:37.6736046Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6736443Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11901 2022-11-23T02:13:37.6736701Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11902 2022-11-23T02:13:37.6737007Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6737444Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6737671Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6738125Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6738357Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6738631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6739060Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6739279Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6739719Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6739959Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6740233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6740708Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6741235Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6741506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6741771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6742582Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:13:37.6742736Z warnings.warn( 2022-11-23T02:13:37.6743601Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:13:37.6743766Z warnings.warn( 2022-11-23T02:13:37.6743909Z ok (5.405s) 2022-11-23T02:13:37.6743916Z 2022-11-23T02:13:37.6744250Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6744403Z Ran 1 test in 5.405s 2022-11-23T02:13:37.6744409Z 2022-11-23T02:13:37.6744543Z OK 2022-11-23T02:13:37.6744549Z 2022-11-23T02:13:37.6744715Z Generating XML reports... 2022-11-23T02:13:37.6745243Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014510.xml 2022-11-23T02:13:37.6745681Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6746111Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6746336Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6746775Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6747002Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6747008Z 2022-11-23T02:13:37.6747157Z Running tests... 2022-11-23T02:13:37.6747483Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6747852Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12108 2022-11-23T02:13:37.6748108Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12109 2022-11-23T02:13:37.6748413Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6748844Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6749002Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6749440Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6749671Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6749950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6750390Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6750608Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6751048Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6751282Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6751714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6752178Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6752625Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6752904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6753296Z STAGE:2022-11-23 01:45:22 12109:12109 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6753565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6753957Z STAGE:2022-11-23 01:45:22 12108:12108 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6754411Z STAGE:2022-11-23 01:45:22 12108:12108 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6754844Z STAGE:2022-11-23 01:45:22 12108:12108 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6755237Z STAGE:2022-11-23 01:45:22 12109:12109 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6755656Z STAGE:2022-11-23 01:45:22 12109:12109 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6756043Z STAGE:2022-11-23 01:45:22 12108:12108 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6756429Z STAGE:2022-11-23 01:45:22 12109:12109 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6756828Z STAGE:2022-11-23 01:45:22 12108:12108 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6757294Z STAGE:2022-11-23 01:45:22 12108:12108 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6757629Z STAGE:2022-11-23 01:45:22 12109:12109 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6758013Z STAGE:2022-11-23 01:45:22 12109:12109 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6758838Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:13:37.6759036Z warnings.warn( 2022-11-23T02:13:37.6759846Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:13:37.6759998Z warnings.warn( 2022-11-23T02:13:37.6760147Z ok (5.008s) 2022-11-23T02:13:37.6760153Z 2022-11-23T02:13:37.6760491Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6760652Z Ran 1 test in 5.008s 2022-11-23T02:13:37.6760658Z 2022-11-23T02:13:37.6760799Z OK 2022-11-23T02:13:37.6760805Z 2022-11-23T02:13:37.6760974Z Generating XML reports... 2022-11-23T02:13:37.6761492Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014519.xml 2022-11-23T02:13:37.6761869Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6762300Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6762519Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6762961Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6763259Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6763338Z 2022-11-23T02:13:37.6763495Z Running tests... 2022-11-23T02:13:37.6763843Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6764214Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12321 2022-11-23T02:13:37.6764472Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12322 2022-11-23T02:13:37.6764780Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6765219Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6765435Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6765881Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6766183Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6766400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6766840Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6767062Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6767500Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6767782Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6768064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6768589Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6769050Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6769329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6769722Z STAGE:2022-11-23 01:45:31 12322:12322 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6769985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6770376Z STAGE:2022-11-23 01:45:31 12321:12321 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6770772Z STAGE:2022-11-23 01:45:31 12321:12321 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6771182Z STAGE:2022-11-23 01:45:31 12321:12321 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6771575Z STAGE:2022-11-23 01:45:31 12322:12322 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6772004Z STAGE:2022-11-23 01:45:31 12322:12322 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6772390Z STAGE:2022-11-23 01:45:31 12321:12321 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6772773Z STAGE:2022-11-23 01:45:31 12322:12322 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6773165Z STAGE:2022-11-23 01:45:31 12322:12322 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6773558Z STAGE:2022-11-23 01:45:31 12321:12321 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6773965Z STAGE:2022-11-23 01:45:31 12321:12321 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6774371Z STAGE:2022-11-23 01:45:31 12322:12322 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6775211Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:13:37.6821167Z warnings.warn( 2022-11-23T02:13:37.6822278Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:13:37.6822383Z warnings.warn( 2022-11-23T02:13:37.6822476Z ok (5.008s) 2022-11-23T02:13:37.6822484Z 2022-11-23T02:13:37.6822766Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6822871Z Ran 1 test in 5.008s 2022-11-23T02:13:37.6822877Z 2022-11-23T02:13:37.6822961Z OK 2022-11-23T02:13:37.6822967Z 2022-11-23T02:13:37.6823083Z Generating XML reports... 2022-11-23T02:13:37.6823993Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014528.xml 2022-11-23T02:13:37.6824321Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6824696Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6824860Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6825247Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6825421Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6825427Z 2022-11-23T02:13:37.6825520Z Running tests... 2022-11-23T02:13:37.6825783Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6826099Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12534 2022-11-23T02:13:37.6826302Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12535 2022-11-23T02:13:37.6826545Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6826919Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6827082Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6827463Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6827643Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6827873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6828276Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6828650Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6828811Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6829199Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6829373Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6829591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6829983Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6830197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6830533Z STAGE:2022-11-23 01:45:40 12535:12535 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6830833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6831174Z STAGE:2022-11-23 01:45:40 12534:12534 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6831516Z STAGE:2022-11-23 01:45:40 12535:12535 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6831857Z STAGE:2022-11-23 01:45:40 12534:12534 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6832212Z STAGE:2022-11-23 01:45:40 12535:12535 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6832565Z STAGE:2022-11-23 01:45:40 12534:12534 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6832895Z STAGE:2022-11-23 01:45:40 12535:12535 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6833296Z STAGE:2022-11-23 01:45:40 12534:12534 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6833629Z STAGE:2022-11-23 01:45:40 12535:12535 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6833968Z STAGE:2022-11-23 01:45:40 12534:12534 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6834319Z STAGE:2022-11-23 01:45:40 12535:12535 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6834671Z STAGE:2022-11-23 01:45:40 12534:12534 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6835423Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:13:37.6835528Z warnings.warn( 2022-11-23T02:13:37.6836276Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:13:37.6836381Z warnings.warn( 2022-11-23T02:13:37.6836473Z ok (5.215s) 2022-11-23T02:13:37.6836479Z 2022-11-23T02:13:37.6836746Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6836843Z Ran 1 test in 5.215s 2022-11-23T02:13:37.6836849Z 2022-11-23T02:13:37.6836933Z OK 2022-11-23T02:13:37.6836939Z 2022-11-23T02:13:37.6837048Z Generating XML reports... 2022-11-23T02:13:37.6837491Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014538.xml 2022-11-23T02:13:37.6837801Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6838179Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6838352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6838739Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6838918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6838924Z 2022-11-23T02:13:37.6839025Z Running tests... 2022-11-23T02:13:37.6839295Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6839621Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12747 2022-11-23T02:13:37.6839832Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12748 2022-11-23T02:13:37.6840088Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6840526Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6840678Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6841058Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6841239Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6841468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6841869Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6842238Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6842399Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6842838Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6843018Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6843246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6843641Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6843858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6844068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6844161Z ok (4.810s) 2022-11-23T02:13:37.6844167Z 2022-11-23T02:13:37.6844433Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6844533Z Ran 1 test in 4.810s 2022-11-23T02:13:37.6844543Z 2022-11-23T02:13:37.6844633Z OK 2022-11-23T02:13:37.6844639Z 2022-11-23T02:13:37.6844753Z Generating XML reports... 2022-11-23T02:13:37.6845198Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014547.xml 2022-11-23T02:13:37.6845511Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6845881Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6846044Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6846412Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6846585Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6846590Z 2022-11-23T02:13:37.6846684Z Running tests... 2022-11-23T02:13:37.6846952Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6847266Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12954 2022-11-23T02:13:37.6847472Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12955 2022-11-23T02:13:37.6847799Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6848175Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6848335Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6848718Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6848896Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6849124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6849591Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6849957Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6850119Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6850498Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6850674Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6850901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6851294Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6851561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6851779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6852001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:13:37.6852219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:13:37.6852605Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.6852991Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.6853321Z STAGE:2022-11-23 01:45:59 12955:12955 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6853644Z STAGE:2022-11-23 01:45:59 12954:12954 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6853985Z STAGE:2022-11-23 01:45:59 12954:12954 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6854543Z STAGE:2022-11-23 01:45:59 12955:12955 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:45:59 12954:12954 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6854550Z 2022-11-23T02:13:37.6854896Z STAGE:2022-11-23 01:45:59 12955:12955 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6855218Z STAGE:2022-11-23 01:45:59 12954:12954 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6855541Z STAGE:2022-11-23 01:45:59 12955:12955 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6855867Z STAGE:2022-11-23 01:45:59 12955:12955 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6856211Z STAGE:2022-11-23 01:45:59 12955:12955 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6856547Z STAGE:2022-11-23 01:45:59 12954:12954 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6856893Z STAGE:2022-11-23 01:45:59 12954:12954 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6856983Z ok (4.931s) 2022-11-23T02:13:37.6856989Z 2022-11-23T02:13:37.6857255Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6857354Z Ran 1 test in 4.932s 2022-11-23T02:13:37.6857360Z 2022-11-23T02:13:37.6857443Z OK 2022-11-23T02:13:37.6857449Z 2022-11-23T02:13:37.6857561Z Generating XML reports... 2022-11-23T02:13:37.6858008Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014556.xml 2022-11-23T02:13:37.6858326Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6858702Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6858922Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6859310Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6859487Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6859493Z 2022-11-23T02:13:37.6859595Z Running tests... 2022-11-23T02:13:37.6859849Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6860156Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13173 2022-11-23T02:13:37.6860360Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13174 2022-11-23T02:13:37.6860616Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6861043Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6861210Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6861596Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6861776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6862003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6862377Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6862542Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6862924Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6863112Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6863328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6863728Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6864115Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6864328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6864536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6864752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:13:37.6864968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:13:37.6865367Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.6865688Z STAGE:2022-11-23 01:46:08 13173:13173 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6866119Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.6866501Z STAGE:2022-11-23 01:46:08 13174:13174 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6866885Z STAGE:2022-11-23 01:46:08 13173:13173 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6867279Z STAGE:2022-11-23 01:46:08 13174:13174 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6867697Z STAGE:2022-11-23 01:46:08 13173:13173 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6868118Z STAGE:2022-11-23 01:46:08 13174:13174 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6868572Z STAGE:2022-11-23 01:46:08 13173:13173 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6868946Z STAGE:2022-11-23 01:46:08 13174:13174 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6869349Z STAGE:2022-11-23 01:46:08 13174:13174 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6869740Z STAGE:2022-11-23 01:46:08 13173:13173 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6870160Z STAGE:2022-11-23 01:46:08 13174:13174 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6870577Z STAGE:2022-11-23 01:46:08 13173:13173 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6870689Z ok (5.015s) 2022-11-23T02:13:37.6870696Z 2022-11-23T02:13:37.6871013Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6871195Z Ran 1 test in 5.015s 2022-11-23T02:13:37.6871203Z 2022-11-23T02:13:37.6871309Z OK 2022-11-23T02:13:37.6871316Z 2022-11-23T02:13:37.6871450Z Generating XML reports... 2022-11-23T02:13:37.6871986Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014605.xml 2022-11-23T02:13:37.6872361Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6872817Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6873011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6873470Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6873685Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6873692Z 2022-11-23T02:13:37.6873823Z Running tests... 2022-11-23T02:13:37.6874140Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6874503Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13392 2022-11-23T02:13:37.6874756Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13393 2022-11-23T02:13:37.6875061Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6875521Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6875695Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6876076Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6876259Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6876487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6876856Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6877022Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6877403Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6877583Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6877814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6878214Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6878611Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6878889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6879106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6879328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:13:37.6879554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:13:37.6879948Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.6880345Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.6880679Z STAGE:2022-11-23 01:46:17 13392:13392 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6881060Z STAGE:2022-11-23 01:46:17 13393:13393 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6881405Z STAGE:2022-11-23 01:46:17 13392:13392 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6881958Z STAGE:2022-11-23 01:46:17 13393:13393 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:46:17 13392:13392 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6881983Z 2022-11-23T02:13:37.6882333Z STAGE:2022-11-23 01:46:17 13393:13393 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6882650Z STAGE:2022-11-23 01:46:17 13392:13392 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6882973Z STAGE:2022-11-23 01:46:17 13393:13393 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6883310Z STAGE:2022-11-23 01:46:17 13392:13392 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6883649Z STAGE:2022-11-23 01:46:17 13393:13393 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6883997Z STAGE:2022-11-23 01:46:17 13392:13392 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6884348Z STAGE:2022-11-23 01:46:17 13393:13393 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6884441Z ok (5.311s) 2022-11-23T02:13:37.6884447Z 2022-11-23T02:13:37.6884713Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6884810Z Ran 1 test in 5.311s 2022-11-23T02:13:37.6884817Z 2022-11-23T02:13:37.6884902Z OK 2022-11-23T02:13:37.6884907Z 2022-11-23T02:13:37.6885023Z Generating XML reports... 2022-11-23T02:13:37.6885465Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014614.xml 2022-11-23T02:13:37.6885786Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6886170Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6886332Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6886716Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6886895Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6886902Z 2022-11-23T02:13:37.6887002Z Running tests... 2022-11-23T02:13:37.6887268Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6887575Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13611 2022-11-23T02:13:37.6887827Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13612 2022-11-23T02:13:37.6888093Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6888553Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6888706Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6889088Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6889263Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6889498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6889870Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6890036Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6890469Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6890653Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6890875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6891277Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6891668Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6891885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6892105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6892326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:13:37.6892561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:13:37.6892955Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.6893338Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.6893668Z STAGE:2022-11-23 01:46:26 13611:13611 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6893991Z STAGE:2022-11-23 01:46:26 13612:13612 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6894328Z STAGE:2022-11-23 01:46:26 13611:13611 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6894679Z STAGE:2022-11-23 01:46:26 13611:13611 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6895018Z STAGE:2022-11-23 01:46:26 13612:13612 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6895370Z STAGE:2022-11-23 01:46:26 13612:13612 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6895698Z STAGE:2022-11-23 01:46:26 13611:13611 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6896014Z STAGE:2022-11-23 01:46:26 13612:13612 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6896350Z STAGE:2022-11-23 01:46:26 13611:13611 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6896708Z STAGE:2022-11-23 01:46:26 13611:13611 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6897042Z STAGE:2022-11-23 01:46:26 13612:13612 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6897391Z STAGE:2022-11-23 01:46:26 13612:13612 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6897485Z ok (5.159s) 2022-11-23T02:13:37.6897544Z 2022-11-23T02:13:37.6897815Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6897918Z Ran 1 test in 5.159s 2022-11-23T02:13:37.6897924Z 2022-11-23T02:13:37.6898005Z OK 2022-11-23T02:13:37.6898010Z 2022-11-23T02:13:37.6898124Z Generating XML reports... 2022-11-23T02:13:37.6898569Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014624.xml 2022-11-23T02:13:37.6898878Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6899249Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6899410Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6899791Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6900015Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6900029Z 2022-11-23T02:13:37.6900124Z Running tests... 2022-11-23T02:13:37.6900392Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6900693Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13830 2022-11-23T02:13:37.6900898Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13831 2022-11-23T02:13:37.6901146Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6901519Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6901671Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6902047Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6902227Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6902455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6902822Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6902984Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6903362Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6903538Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6903759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6904152Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6904542Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6904756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6904962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6905106Z skip: Skipped due to small world size. (5.156s) 2022-11-23T02:13:37.6905112Z 2022-11-23T02:13:37.6905374Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6905470Z Ran 1 test in 5.157s 2022-11-23T02:13:37.6905476Z 2022-11-23T02:13:37.6905570Z OK (skipped=1) 2022-11-23T02:13:37.6905576Z 2022-11-23T02:13:37.6905688Z Generating XML reports... 2022-11-23T02:13:37.6906127Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014633.xml 2022-11-23T02:13:37.6906441Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6906883Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6907043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6907410Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6907585Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6907603Z 2022-11-23T02:13:37.6907691Z Running tests... 2022-11-23T02:13:37.6907954Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6908255Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14037 2022-11-23T02:13:37.6908458Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14038 2022-11-23T02:13:37.6908756Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6909125Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6909286Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6909666Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6909842Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6910063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6910428Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6910589Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6910971Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6911149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6911374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6911767Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6912158Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6912367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6912579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6912723Z skip: Skipped due to small world size. (4.904s) 2022-11-23T02:13:37.6912729Z 2022-11-23T02:13:37.6913003Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6913091Z Ran 1 test in 4.905s 2022-11-23T02:13:37.6913104Z 2022-11-23T02:13:37.6913189Z OK (skipped=1) 2022-11-23T02:13:37.6913195Z 2022-11-23T02:13:37.6913306Z Generating XML reports... 2022-11-23T02:13:37.6913748Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014642.xml 2022-11-23T02:13:37.6914057Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6914425Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6914584Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6914962Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6915137Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6915202Z 2022-11-23T02:13:37.6915305Z Running tests... 2022-11-23T02:13:37.6915572Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6915878Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14244 2022-11-23T02:13:37.6916082Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14245 2022-11-23T02:13:37.6916328Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6916693Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6916851Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6917225Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6917454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6917675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6918069Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6918435Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6918597Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6918975Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6919139Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6919357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6919752Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6919966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6920174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6920318Z skip: Skipped due to small world size. (4.907s) 2022-11-23T02:13:37.6920324Z 2022-11-23T02:13:37.6920586Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6920683Z Ran 1 test in 4.907s 2022-11-23T02:13:37.6920689Z 2022-11-23T02:13:37.6920782Z OK (skipped=1) 2022-11-23T02:13:37.6920787Z 2022-11-23T02:13:37.6920898Z Generating XML reports... 2022-11-23T02:13:37.6921340Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014651.xml 2022-11-23T02:13:37.6921652Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6922025Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6922185Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6922560Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6922734Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6922740Z 2022-11-23T02:13:37.6922837Z Running tests... 2022-11-23T02:13:37.6923101Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6923401Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14451 2022-11-23T02:13:37.6923602Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14452 2022-11-23T02:13:37.6923852Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6924277Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6924438Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6924808Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6924980Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6925197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6925562Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6925724Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6926191Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6926375Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6926596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6926994Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6927382Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6927591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6927931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6928076Z skip: Skipped due to small world size. (4.808s) 2022-11-23T02:13:37.6928082Z 2022-11-23T02:13:37.6928353Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6928454Z Ran 1 test in 4.808s 2022-11-23T02:13:37.6928459Z 2022-11-23T02:13:37.6928553Z OK (skipped=1) 2022-11-23T02:13:37.6928558Z 2022-11-23T02:13:37.6928668Z Generating XML reports... 2022-11-23T02:13:37.6929113Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014700.xml 2022-11-23T02:13:37.6929422Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6929791Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6929953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6930333Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6930498Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6930519Z 2022-11-23T02:13:37.6930607Z Running tests... 2022-11-23T02:13:37.6930870Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6931163Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14658 2022-11-23T02:13:37.6931368Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14659 2022-11-23T02:13:37.6931619Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6931984Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6932144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6932522Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6932777Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6932996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6933367Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6933529Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6933905Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6934079Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6934296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6934689Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6935126Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6935351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6935685Z STAGE:2022-11-23 01:47:12 14659:14659 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6935895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6936221Z STAGE:2022-11-23 01:47:12 14658:14658 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6936555Z STAGE:2022-11-23 01:47:12 14659:14659 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6936891Z STAGE:2022-11-23 01:47:12 14659:14659 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6937225Z STAGE:2022-11-23 01:47:12 14658:14658 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6937577Z STAGE:2022-11-23 01:47:12 14658:14658 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6937900Z STAGE:2022-11-23 01:47:12 14659:14659 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6938225Z STAGE:2022-11-23 01:47:12 14658:14658 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6938555Z STAGE:2022-11-23 01:47:12 14659:14659 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6938902Z STAGE:2022-11-23 01:47:12 14659:14659 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6939234Z STAGE:2022-11-23 01:47:12 14658:14658 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6939578Z STAGE:2022-11-23 01:47:12 14658:14658 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6939672Z ok (5.617s) 2022-11-23T02:13:37.6939678Z 2022-11-23T02:13:37.6939947Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6940047Z Ran 1 test in 5.617s 2022-11-23T02:13:37.6940053Z 2022-11-23T02:13:37.6940133Z OK 2022-11-23T02:13:37.6940139Z 2022-11-23T02:13:37.6940249Z Generating XML reports... 2022-11-23T02:13:37.6940691Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014709.xml 2022-11-23T02:13:37.6941002Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6941374Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6941535Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6941919Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6942092Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6942156Z 2022-11-23T02:13:37.6942257Z Running tests... 2022-11-23T02:13:37.6942525Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6942817Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14871 2022-11-23T02:13:37.6943012Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14872 2022-11-23T02:13:37.6943266Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6943631Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6943789Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6944166Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6944385Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6944611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6944979Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6945139Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6945514Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6945691Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6945912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6946305Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6946698Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6946909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6947121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6947449Z STAGE:2022-11-23 01:47:22 14871:14871 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6947781Z STAGE:2022-11-23 01:47:22 14872:14872 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6948116Z STAGE:2022-11-23 01:47:22 14871:14871 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6948464Z STAGE:2022-11-23 01:47:22 14871:14871 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6948800Z STAGE:2022-11-23 01:47:22 14872:14872 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6949155Z STAGE:2022-11-23 01:47:22 14872:14872 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6949490Z STAGE:2022-11-23 01:47:22 14871:14871 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6949804Z STAGE:2022-11-23 01:47:22 14872:14872 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6950134Z STAGE:2022-11-23 01:47:22 14871:14871 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6950464Z STAGE:2022-11-23 01:47:22 14872:14872 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6950811Z STAGE:2022-11-23 01:47:22 14871:14871 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6951159Z STAGE:2022-11-23 01:47:22 14872:14872 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6951254Z ok (4.871s) 2022-11-23T02:13:37.6951261Z 2022-11-23T02:13:37.6951526Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6951681Z Ran 1 test in 4.872s 2022-11-23T02:13:37.6951687Z 2022-11-23T02:13:37.6951769Z OK 2022-11-23T02:13:37.6951775Z 2022-11-23T02:13:37.6951886Z Generating XML reports... 2022-11-23T02:13:37.6952325Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014719.xml 2022-11-23T02:13:37.6952635Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6953002Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6953164Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6953544Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6953718Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6953724Z 2022-11-23T02:13:37.6953880Z Running tests... 2022-11-23T02:13:37.6954157Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6954459Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15084 2022-11-23T02:13:37.6954667Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15085 2022-11-23T02:13:37.6954914Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6955283Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6955444Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6955815Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6955992Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6956220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6956585Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6956750Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6957136Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6957317Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6957541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6957934Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6958335Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6958550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6958770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6959102Z STAGE:2022-11-23 01:47:31 15085:15085 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6959428Z STAGE:2022-11-23 01:47:31 15084:15084 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6959761Z STAGE:2022-11-23 01:47:31 15084:15084 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6960107Z STAGE:2022-11-23 01:47:31 15084:15084 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6960331Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:13:37.6960669Z STAGE:2022-11-23 01:47:31 15085:15085 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6961075Z STAGE:2022-11-23 01:47:31 15085:15085 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6961294Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:13:37.6961619Z STAGE:2022-11-23 01:47:31 15084:15084 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6961941Z STAGE:2022-11-23 01:47:31 15085:15085 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6962269Z STAGE:2022-11-23 01:47:31 15085:15085 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6962836Z STAGE:2022-11-23 01:47:31 15085:15085 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:47:31 15084:15084 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6962843Z 2022-11-23T02:13:37.6963241Z STAGE:2022-11-23 01:47:31 15084:15084 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6964038Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1506: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2022-11-23T02:13:37.6964138Z warnings.warn( 2022-11-23T02:13:37.6964895Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1506: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2022-11-23T02:13:37.6964985Z warnings.warn( 2022-11-23T02:13:37.6965077Z ok (5.560s) 2022-11-23T02:13:37.6965083Z 2022-11-23T02:13:37.6965349Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6965454Z Ran 1 test in 5.561s 2022-11-23T02:13:37.6965465Z 2022-11-23T02:13:37.6965547Z OK 2022-11-23T02:13:37.6965553Z 2022-11-23T02:13:37.6965670Z Generating XML reports... 2022-11-23T02:13:37.6966115Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014728.xml 2022-11-23T02:13:37.6966428Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6966805Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6966965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6967345Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6967521Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6967529Z 2022-11-23T02:13:37.6967630Z Running tests... 2022-11-23T02:13:37.6967944Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6968262Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15297 2022-11-23T02:13:37.6968466Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15298 2022-11-23T02:13:37.6968723Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6969088Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6969264Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6969648Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6969825Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6970119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6970478Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6970646Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6971026Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6971204Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6971426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6971833Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6972219Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6972502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6972716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6973050Z STAGE:2022-11-23 01:47:41 15298:15298 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6973375Z STAGE:2022-11-23 01:47:41 15297:15297 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6973707Z STAGE:2022-11-23 01:47:41 15297:15297 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6974057Z STAGE:2022-11-23 01:47:41 15297:15297 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6974278Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:13:37.6974608Z STAGE:2022-11-23 01:47:41 15298:15298 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6974959Z STAGE:2022-11-23 01:47:41 15298:15298 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6975179Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:13:37.6975509Z STAGE:2022-11-23 01:47:41 15297:15297 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6975833Z STAGE:2022-11-23 01:47:41 15298:15298 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6976163Z STAGE:2022-11-23 01:47:41 15298:15298 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6976506Z STAGE:2022-11-23 01:47:41 15298:15298 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6976838Z STAGE:2022-11-23 01:47:41 15297:15297 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6977189Z STAGE:2022-11-23 01:47:41 15297:15297 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6977955Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1506: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2022-11-23T02:13:37.6978055Z warnings.warn( 2022-11-23T02:13:37.6978808Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1506: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2022-11-23T02:13:37.6978905Z warnings.warn( 2022-11-23T02:13:37.6978986Z ok (5.318s) 2022-11-23T02:13:37.6978991Z 2022-11-23T02:13:37.6979255Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6979351Z Ran 1 test in 5.319s 2022-11-23T02:13:37.6979411Z 2022-11-23T02:13:37.6979497Z OK 2022-11-23T02:13:37.6979503Z 2022-11-23T02:13:37.6979616Z Generating XML reports... 2022-11-23T02:13:37.6980056Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014738.xml 2022-11-23T02:13:37.6980367Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6980735Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6980896Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6981273Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6981448Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6981454Z 2022-11-23T02:13:37.6981551Z Running tests... 2022-11-23T02:13:37.6981862Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6982167Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15510 2022-11-23T02:13:37.6982369Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15511 2022-11-23T02:13:37.6982621Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6982991Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6983154Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6983530Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6983704Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6983930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6984316Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6984678Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6984838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6985217Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6985389Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6985606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6985997Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6986214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6986541Z STAGE:2022-11-23 01:47:50 15511:15511 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6986753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6987081Z STAGE:2022-11-23 01:47:50 15510:15510 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6987415Z STAGE:2022-11-23 01:47:50 15510:15510 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6987762Z STAGE:2022-11-23 01:47:50 15510:15510 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6988095Z STAGE:2022-11-23 01:47:50 15511:15511 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6988440Z STAGE:2022-11-23 01:47:50 15511:15511 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6988825Z STAGE:2022-11-23 01:47:50 15510:15510 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6989146Z STAGE:2022-11-23 01:47:50 15511:15511 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.6989477Z STAGE:2022-11-23 01:47:50 15510:15510 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6989824Z STAGE:2022-11-23 01:47:50 15510:15510 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6990157Z STAGE:2022-11-23 01:47:50 15511:15511 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.6990504Z STAGE:2022-11-23 01:47:50 15511:15511 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.6990595Z ok (5.312s) 2022-11-23T02:13:37.6990601Z 2022-11-23T02:13:37.6990865Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6990953Z Ran 1 test in 5.312s 2022-11-23T02:13:37.6990968Z 2022-11-23T02:13:37.6991045Z OK 2022-11-23T02:13:37.6991099Z 2022-11-23T02:13:37.6991217Z Generating XML reports... 2022-11-23T02:13:37.6991656Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014747.xml 2022-11-23T02:13:37.6991965Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6992333Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6992494Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6992873Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6993048Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6993054Z 2022-11-23T02:13:37.6993150Z Running tests... 2022-11-23T02:13:37.6993413Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6993723Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15723 2022-11-23T02:13:37.6993925Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15724 2022-11-23T02:13:37.6994178Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.6994546Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6994706Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6995085Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6995262Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6995484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.6995853Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.6996014Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.6996391Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.6996568Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.6996777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.6997167Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6997559Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.6997775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.6998042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.6998130Z ok (5.315s) 2022-11-23T02:13:37.6998136Z 2022-11-23T02:13:37.6998403Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.6998503Z Ran 1 test in 5.315s 2022-11-23T02:13:37.6998509Z 2022-11-23T02:13:37.6998590Z OK 2022-11-23T02:13:37.6998596Z 2022-11-23T02:13:37.6998706Z Generating XML reports... 2022-11-23T02:13:37.6999145Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014756.xml 2022-11-23T02:13:37.6999454Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.6999823Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7000031Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7000420Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7000594Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7000600Z 2022-11-23T02:13:37.7000695Z Running tests... 2022-11-23T02:13:37.7000956Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7001248Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15930 2022-11-23T02:13:37.7001452Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15931 2022-11-23T02:13:37.7001702Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7002074Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7002230Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7002607Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7002781Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7003007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7003372Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7003530Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7003907Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7004082Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7004310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7004702Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7005091Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7005303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7005629Z STAGE:2022-11-23 01:48:09 15931:15931 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7005840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7006168Z STAGE:2022-11-23 01:48:09 15930:15930 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7006720Z STAGE:2022-11-23 01:48:09 15930:15930 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:48:09 15931:15931 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7006780Z 2022-11-23T02:13:37.7007360Z STAGE:2022-11-23 01:48:09 15930:15930 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:48:09 15931:15931 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7007366Z 2022-11-23T02:13:37.7007750Z STAGE:2022-11-23 01:48:09 15931:15931 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7008078Z STAGE:2022-11-23 01:48:09 15930:15930 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7008411Z STAGE:2022-11-23 01:48:09 15930:15930 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7008970Z STAGE:2022-11-23 01:48:09 15931:15931 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:48:09 15930:15930 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7009039Z 2022-11-23T02:13:37.7009391Z STAGE:2022-11-23 01:48:09 15931:15931 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7009480Z ok (4.911s) 2022-11-23T02:13:37.7009486Z 2022-11-23T02:13:37.7009750Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7009847Z Ran 1 test in 4.911s 2022-11-23T02:13:37.7009853Z 2022-11-23T02:13:37.7009935Z OK 2022-11-23T02:13:37.7009941Z 2022-11-23T02:13:37.7010052Z Generating XML reports... 2022-11-23T02:13:37.7010493Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014806.xml 2022-11-23T02:13:37.7010802Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7011161Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7011323Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7011705Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7011880Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7011886Z 2022-11-23T02:13:37.7011982Z Running tests... 2022-11-23T02:13:37.7012244Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7012541Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16143 2022-11-23T02:13:37.7012743Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16144 2022-11-23T02:13:37.7012991Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7013362Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7013528Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7013905Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7014077Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7014298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7014662Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7014821Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7015201Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7015377Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7015600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7016059Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7016447Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7016660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7016986Z STAGE:2022-11-23 01:48:18 16144:16144 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7017187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7017512Z STAGE:2022-11-23 01:48:18 16143:16143 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7017851Z STAGE:2022-11-23 01:48:18 16143:16143 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7018254Z STAGE:2022-11-23 01:48:18 16143:16143 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7018592Z STAGE:2022-11-23 01:48:18 16144:16144 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7018942Z STAGE:2022-11-23 01:48:18 16144:16144 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7019268Z STAGE:2022-11-23 01:48:18 16143:16143 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7019588Z STAGE:2022-11-23 01:48:18 16144:16144 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7020130Z STAGE:2022-11-23 01:48:18 16144:16144 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:48:18 16143:16143 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7020137Z 2022-11-23T02:13:37.7020723Z STAGE:2022-11-23 01:48:18 16144:16144 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:48:18 16143:16143 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7020733Z 2022-11-23T02:13:37.7020826Z ok (5.016s) 2022-11-23T02:13:37.7020832Z 2022-11-23T02:13:37.7021097Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7021197Z Ran 1 test in 5.016s 2022-11-23T02:13:37.7021203Z 2022-11-23T02:13:37.7021284Z OK 2022-11-23T02:13:37.7021290Z 2022-11-23T02:13:37.7021401Z Generating XML reports... 2022-11-23T02:13:37.7021840Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014815.xml 2022-11-23T02:13:37.7022150Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7022521Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7022684Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7023070Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7023249Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7023254Z 2022-11-23T02:13:37.7023349Z Running tests... 2022-11-23T02:13:37.7023609Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7023917Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16356 2022-11-23T02:13:37.7024120Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16357 2022-11-23T02:13:37.7024373Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7024747Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7024960Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7025340Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7025517Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7025743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7026112Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7026276Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7026656Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7026834Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7027104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7027506Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7027898Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7028117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7028446Z STAGE:2022-11-23 01:48:27 16357:16357 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7028656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7028985Z STAGE:2022-11-23 01:48:27 16356:16356 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7029319Z STAGE:2022-11-23 01:48:27 16356:16356 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7029658Z STAGE:2022-11-23 01:48:27 16357:16357 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7030008Z STAGE:2022-11-23 01:48:27 16356:16356 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7030354Z STAGE:2022-11-23 01:48:27 16357:16357 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7030681Z STAGE:2022-11-23 01:48:27 16356:16356 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7031007Z STAGE:2022-11-23 01:48:27 16357:16357 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7031549Z STAGE:2022-11-23 01:48:27 16356:16356 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:48:27 16357:16357 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7031556Z 2022-11-23T02:13:37.7031910Z STAGE:2022-11-23 01:48:27 16357:16357 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7032262Z STAGE:2022-11-23 01:48:27 16356:16356 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7032354Z ok (4.906s) 2022-11-23T02:13:37.7032360Z 2022-11-23T02:13:37.7032614Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7032719Z Ran 1 test in 4.906s 2022-11-23T02:13:37.7032726Z 2022-11-23T02:13:37.7032808Z OK 2022-11-23T02:13:37.7032813Z 2022-11-23T02:13:37.7032931Z Generating XML reports... 2022-11-23T02:13:37.7033375Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014824.xml 2022-11-23T02:13:37.7033691Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7034060Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7034229Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7034668Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7034844Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7034851Z 2022-11-23T02:13:37.7034952Z Running tests... 2022-11-23T02:13:37.7035217Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7035520Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16569 2022-11-23T02:13:37.7035726Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16570 2022-11-23T02:13:37.7035979Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7036350Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7036568Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7036963Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7037143Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7037371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7037737Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7037906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7038274Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7038451Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7038684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7039086Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7039478Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7039699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7039912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7040249Z STAGE:2022-11-23 01:48:36 16570:16570 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7040579Z STAGE:2022-11-23 01:48:36 16569:16569 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7040920Z STAGE:2022-11-23 01:48:36 16569:16569 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7041275Z STAGE:2022-11-23 01:48:36 16569:16569 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7041501Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:13:37.7041841Z STAGE:2022-11-23 01:48:36 16570:16570 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7042189Z STAGE:2022-11-23 01:48:36 16570:16570 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7042411Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:13:37.7042739Z STAGE:2022-11-23 01:48:36 16569:16569 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7043066Z STAGE:2022-11-23 01:48:36 16570:16570 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7043406Z STAGE:2022-11-23 01:48:36 16569:16569 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7043834Z STAGE:2022-11-23 01:48:36 16569:16569 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7044177Z STAGE:2022-11-23 01:48:36 16570:16570 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7044530Z STAGE:2022-11-23 01:48:36 16570:16570 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7044862Z STAGE:2022-11-23 01:48:36 16569:16569 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7045184Z STAGE:2022-11-23 01:48:36 16570:16570 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7045520Z STAGE:2022-11-23 01:48:36 16570:16570 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7045844Z STAGE:2022-11-23 01:48:36 16569:16569 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7046186Z STAGE:2022-11-23 01:48:36 16570:16570 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7046582Z STAGE:2022-11-23 01:48:36 16569:16569 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7046679Z ok (5.624s) 2022-11-23T02:13:37.7046685Z 2022-11-23T02:13:37.7046961Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7047061Z Ran 1 test in 5.625s 2022-11-23T02:13:37.7047067Z 2022-11-23T02:13:37.7047149Z OK 2022-11-23T02:13:37.7047155Z 2022-11-23T02:13:37.7047268Z Generating XML reports... 2022-11-23T02:13:37.7047880Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014833.xml 2022-11-23T02:13:37.7048201Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7048571Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7048742Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7049136Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7049311Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7049317Z 2022-11-23T02:13:37.7049414Z Running tests... 2022-11-23T02:13:37.7049683Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7049991Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16782 2022-11-23T02:13:37.7050194Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16783 2022-11-23T02:13:37.7050448Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7050815Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7050983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7051362Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7051527Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7051754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7052119Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7052282Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7052661Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7052834Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7053063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7053539Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7053924Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7054136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7054346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7054678Z STAGE:2022-11-23 01:48:46 16783:16783 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7055000Z STAGE:2022-11-23 01:48:46 16782:16782 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7055332Z STAGE:2022-11-23 01:48:46 16782:16782 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7055733Z STAGE:2022-11-23 01:48:46 16782:16782 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7055957Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:13:37.7056291Z STAGE:2022-11-23 01:48:46 16783:16783 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7056634Z STAGE:2022-11-23 01:48:46 16783:16783 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7056854Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:13:37.7057181Z STAGE:2022-11-23 01:48:46 16782:16782 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7057506Z STAGE:2022-11-23 01:48:46 16783:16783 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7057838Z STAGE:2022-11-23 01:48:46 16782:16782 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7058190Z STAGE:2022-11-23 01:48:46 16782:16782 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7058524Z STAGE:2022-11-23 01:48:46 16783:16783 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7058857Z STAGE:2022-11-23 01:48:46 16783:16783 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7059182Z STAGE:2022-11-23 01:48:46 16782:16782 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7059504Z STAGE:2022-11-23 01:48:46 16783:16783 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7059835Z STAGE:2022-11-23 01:48:46 16783:16783 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7060181Z STAGE:2022-11-23 01:48:46 16783:16783 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7060513Z STAGE:2022-11-23 01:48:46 16782:16782 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7060864Z STAGE:2022-11-23 01:48:46 16782:16782 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7060952Z ok (5.606s) 2022-11-23T02:13:37.7060958Z 2022-11-23T02:13:37.7061220Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7061316Z Ran 1 test in 5.607s 2022-11-23T02:13:37.7061324Z 2022-11-23T02:13:37.7061404Z OK 2022-11-23T02:13:37.7061410Z 2022-11-23T02:13:37.7061521Z Generating XML reports... 2022-11-23T02:13:37.7061958Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014843.xml 2022-11-23T02:13:37.7062267Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7062636Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7062797Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7063243Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7063418Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7063424Z 2022-11-23T02:13:37.7063519Z Running tests... 2022-11-23T02:13:37.7063782Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7064092Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16995 2022-11-23T02:13:37.7064297Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16996 2022-11-23T02:13:37.7064538Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7064905Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7065121Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7065508Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7065683Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7065903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7066270Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7066433Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7066815Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7066989Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7067214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7067612Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7068001Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7068213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7068423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7068754Z STAGE:2022-11-23 01:48:55 16996:16996 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7069078Z STAGE:2022-11-23 01:48:56 16995:16995 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7069413Z STAGE:2022-11-23 01:48:56 16995:16995 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7069764Z STAGE:2022-11-23 01:48:56 16995:16995 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7069986Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:13:37.7070319Z STAGE:2022-11-23 01:48:56 16996:16996 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7070664Z STAGE:2022-11-23 01:48:56 16996:16996 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7070880Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:13:37.7071207Z STAGE:2022-11-23 01:48:56 16995:16995 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7071520Z STAGE:2022-11-23 01:48:56 16996:16996 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7071850Z STAGE:2022-11-23 01:48:56 16996:16996 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7072254Z STAGE:2022-11-23 01:48:56 16996:16996 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7072588Z STAGE:2022-11-23 01:48:56 16995:16995 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7072932Z STAGE:2022-11-23 01:48:56 16995:16995 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7073257Z STAGE:2022-11-23 01:48:56 16996:16996 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7073580Z STAGE:2022-11-23 01:48:56 16995:16995 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7073914Z STAGE:2022-11-23 01:48:56 16995:16995 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7074263Z STAGE:2022-11-23 01:48:56 16995:16995 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7074594Z STAGE:2022-11-23 01:48:56 16996:16996 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7074986Z STAGE:2022-11-23 01:48:56 16996:16996 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7075078Z ok (5.762s) 2022-11-23T02:13:37.7075084Z 2022-11-23T02:13:37.7075351Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7075447Z Ran 1 test in 5.762s 2022-11-23T02:13:37.7075452Z 2022-11-23T02:13:37.7075533Z OK 2022-11-23T02:13:37.7075538Z 2022-11-23T02:13:37.7075650Z Generating XML reports... 2022-11-23T02:13:37.7076090Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014852.xml 2022-11-23T02:13:37.7076400Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7076770Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7076933Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7077318Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7077494Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7077501Z 2022-11-23T02:13:37.7077597Z Running tests... 2022-11-23T02:13:37.7077852Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7078074Z test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s) 2022-11-23T02:13:37.7078080Z 2022-11-23T02:13:37.7078342Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7078439Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7078445Z 2022-11-23T02:13:37.7078540Z OK (skipped=1) 2022-11-23T02:13:37.7078545Z 2022-11-23T02:13:37.7078659Z Generating XML reports... 2022-11-23T02:13:37.7079104Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014902.xml 2022-11-23T02:13:37.7079419Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7079789Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7079952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7080332Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7080509Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7080515Z 2022-11-23T02:13:37.7080611Z Running tests... 2022-11-23T02:13:37.7080874Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7081110Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s) 2022-11-23T02:13:37.7081116Z 2022-11-23T02:13:37.7081440Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7081537Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7081542Z 2022-11-23T02:13:37.7081636Z OK (skipped=1) 2022-11-23T02:13:37.7081642Z 2022-11-23T02:13:37.7081755Z Generating XML reports... 2022-11-23T02:13:37.7082192Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014906.xml 2022-11-23T02:13:37.7082504Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7082876Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7083028Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7083408Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7083582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7083641Z 2022-11-23T02:13:37.7083741Z Running tests... 2022-11-23T02:13:37.7084009Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7084249Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s) 2022-11-23T02:13:37.7084255Z 2022-11-23T02:13:37.7084514Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7084610Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7084616Z 2022-11-23T02:13:37.7084709Z OK (skipped=1) 2022-11-23T02:13:37.7084715Z 2022-11-23T02:13:37.7084827Z Generating XML reports... 2022-11-23T02:13:37.7085266Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014910.xml 2022-11-23T02:13:37.7085577Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7085952Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7086120Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7086502Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7086680Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7086686Z 2022-11-23T02:13:37.7086782Z Running tests... 2022-11-23T02:13:37.7087045Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7087296Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s) 2022-11-23T02:13:37.7087302Z 2022-11-23T02:13:37.7087561Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7087657Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7087663Z 2022-11-23T02:13:37.7087803Z OK (skipped=1) 2022-11-23T02:13:37.7087812Z 2022-11-23T02:13:37.7087929Z Generating XML reports... 2022-11-23T02:13:37.7088355Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014914.xml 2022-11-23T02:13:37.7088668Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7089036Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7089199Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7089580Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7089757Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7089763Z 2022-11-23T02:13:37.7089860Z Running tests... 2022-11-23T02:13:37.7090122Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7090431Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s) 2022-11-23T02:13:37.7090438Z 2022-11-23T02:13:37.7090702Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7090799Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7090804Z 2022-11-23T02:13:37.7090900Z OK (skipped=1) 2022-11-23T02:13:37.7090906Z 2022-11-23T02:13:37.7091018Z Generating XML reports... 2022-11-23T02:13:37.7091463Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014918.xml 2022-11-23T02:13:37.7091772Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7092139Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7092298Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7092730Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7092917Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7092923Z 2022-11-23T02:13:37.7093021Z Running tests... 2022-11-23T02:13:37.7093286Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7093545Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s) 2022-11-23T02:13:37.7093552Z 2022-11-23T02:13:37.7093814Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7093902Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7093908Z 2022-11-23T02:13:37.7094003Z OK (skipped=1) 2022-11-23T02:13:37.7094009Z 2022-11-23T02:13:37.7094121Z Generating XML reports... 2022-11-23T02:13:37.7094561Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014922.xml 2022-11-23T02:13:37.7094878Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7095250Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7095411Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7095789Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7095966Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7095972Z 2022-11-23T02:13:37.7096069Z Running tests... 2022-11-23T02:13:37.7096334Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7096568Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s) 2022-11-23T02:13:37.7096574Z 2022-11-23T02:13:37.7096833Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7096938Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7096944Z 2022-11-23T02:13:37.7097038Z OK (skipped=1) 2022-11-23T02:13:37.7097044Z 2022-11-23T02:13:37.7097156Z Generating XML reports... 2022-11-23T02:13:37.7097592Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014926.xml 2022-11-23T02:13:37.7097901Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7098271Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7098430Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7098810Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7098986Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7099048Z 2022-11-23T02:13:37.7099141Z Running tests... 2022-11-23T02:13:37.7099412Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7099671Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-11-23T02:13:37.7099677Z 2022-11-23T02:13:37.7099936Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7100032Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7100038Z 2022-11-23T02:13:37.7100131Z OK (skipped=1) 2022-11-23T02:13:37.7100137Z 2022-11-23T02:13:37.7100248Z Generating XML reports... 2022-11-23T02:13:37.7100687Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014930.xml 2022-11-23T02:13:37.7100999Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7101412Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7101582Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7101964Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7102138Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7102144Z 2022-11-23T02:13:37.7102240Z Running tests... 2022-11-23T02:13:37.7102503Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7102767Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:13:37.7102774Z 2022-11-23T02:13:37.7103034Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7103131Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7103137Z 2022-11-23T02:13:37.7103231Z OK (skipped=1) 2022-11-23T02:13:37.7103236Z 2022-11-23T02:13:37.7103355Z Generating XML reports... 2022-11-23T02:13:37.7103791Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014934.xml 2022-11-23T02:13:37.7104101Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7104468Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7104621Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7104999Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7105175Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7105181Z 2022-11-23T02:13:37.7105277Z Running tests... 2022-11-23T02:13:37.7105540Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7105822Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:13:37.7105832Z 2022-11-23T02:13:37.7106095Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7106193Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7106200Z 2022-11-23T02:13:37.7106293Z OK (skipped=1) 2022-11-23T02:13:37.7106299Z 2022-11-23T02:13:37.7106412Z Generating XML reports... 2022-11-23T02:13:37.7106849Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014938.xml 2022-11-23T02:13:37.7107168Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7107538Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7107703Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7108086Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7108345Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7108351Z 2022-11-23T02:13:37.7108449Z Running tests... 2022-11-23T02:13:37.7108715Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7108987Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-11-23T02:13:37.7108994Z 2022-11-23T02:13:37.7109253Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7109351Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7109357Z 2022-11-23T02:13:37.7109451Z OK (skipped=1) 2022-11-23T02:13:37.7109457Z 2022-11-23T02:13:37.7109572Z Generating XML reports... 2022-11-23T02:13:37.7109995Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014942.xml 2022-11-23T02:13:37.7110377Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7110752Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7110915Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7111297Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7111473Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7111479Z 2022-11-23T02:13:37.7111575Z Running tests... 2022-11-23T02:13:37.7111839Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7112127Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-11-23T02:13:37.7112133Z 2022-11-23T02:13:37.7112403Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7112500Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7112506Z 2022-11-23T02:13:37.7112599Z OK (skipped=1) 2022-11-23T02:13:37.7112604Z 2022-11-23T02:13:37.7112717Z Generating XML reports... 2022-11-23T02:13:37.7113155Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014946.xml 2022-11-23T02:13:37.7113466Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7113836Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7113999Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7114378Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7114555Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7114567Z 2022-11-23T02:13:37.7114663Z Running tests... 2022-11-23T02:13:37.7114926Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7115204Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:13:37.7115210Z 2022-11-23T02:13:37.7115471Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7115559Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7115573Z 2022-11-23T02:13:37.7115659Z OK (skipped=1) 2022-11-23T02:13:37.7115664Z 2022-11-23T02:13:37.7115775Z Generating XML reports... 2022-11-23T02:13:37.7116216Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014950.xml 2022-11-23T02:13:37.7116526Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7116903Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7117192Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7117578Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7117755Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7117760Z 2022-11-23T02:13:37.7117857Z Running tests... 2022-11-23T02:13:37.7118123Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7118409Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-11-23T02:13:37.7118416Z 2022-11-23T02:13:37.7118675Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7118771Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7118777Z 2022-11-23T02:13:37.7118927Z OK (skipped=1) 2022-11-23T02:13:37.7118934Z 2022-11-23T02:13:37.7119047Z Generating XML reports... 2022-11-23T02:13:37.7119484Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014955.xml 2022-11-23T02:13:37.7119796Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7120164Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7120324Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7120704Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7120879Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7120885Z 2022-11-23T02:13:37.7120982Z Running tests... 2022-11-23T02:13:37.7121239Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7121522Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2022-11-23T02:13:37.7121538Z 2022-11-23T02:13:37.7121789Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7121886Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7121892Z 2022-11-23T02:13:37.7121985Z OK (skipped=1) 2022-11-23T02:13:37.7121991Z 2022-11-23T02:13:37.7122103Z Generating XML reports... 2022-11-23T02:13:37.7122538Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014959.xml 2022-11-23T02:13:37.7122850Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7123222Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7123382Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7123766Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7123939Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7123945Z 2022-11-23T02:13:37.7124041Z Running tests... 2022-11-23T02:13:37.7124304Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7124583Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-11-23T02:13:37.7124589Z 2022-11-23T02:13:37.7124851Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7124951Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7124956Z 2022-11-23T02:13:37.7125049Z OK (skipped=1) 2022-11-23T02:13:37.7125054Z 2022-11-23T02:13:37.7125166Z Generating XML reports... 2022-11-23T02:13:37.7125606Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015003.xml 2022-11-23T02:13:37.7125975Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7126349Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7126511Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7126883Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7127059Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7127076Z 2022-11-23T02:13:37.7127164Z Running tests... 2022-11-23T02:13:37.7127426Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7127731Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:13:37.7127799Z 2022-11-23T02:13:37.7128075Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7128171Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7128177Z 2022-11-23T02:13:37.7128270Z OK (skipped=1) 2022-11-23T02:13:37.7128276Z 2022-11-23T02:13:37.7128386Z Generating XML reports... 2022-11-23T02:13:37.7128823Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015007.xml 2022-11-23T02:13:37.7129134Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7129506Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7129669Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7130051Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7130234Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7130240Z 2022-11-23T02:13:37.7130336Z Running tests... 2022-11-23T02:13:37.7130598Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7130878Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:13:37.7130884Z 2022-11-23T02:13:37.7131147Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7131243Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7131249Z 2022-11-23T02:13:37.7131342Z OK (skipped=1) 2022-11-23T02:13:37.7131347Z 2022-11-23T02:13:37.7131460Z Generating XML reports... 2022-11-23T02:13:37.7131896Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015011.xml 2022-11-23T02:13:37.7132212Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7132574Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7132739Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7133121Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7133297Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7133303Z 2022-11-23T02:13:37.7133399Z Running tests... 2022-11-23T02:13:37.7133661Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7133944Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-11-23T02:13:37.7133950Z 2022-11-23T02:13:37.7134212Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7134372Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7134382Z 2022-11-23T02:13:37.7134475Z OK (skipped=1) 2022-11-23T02:13:37.7134481Z 2022-11-23T02:13:37.7134593Z Generating XML reports... 2022-11-23T02:13:37.7135035Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015015.xml 2022-11-23T02:13:37.7135344Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7135712Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7135873Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7136253Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7136425Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7136431Z 2022-11-23T02:13:37.7136527Z Running tests... 2022-11-23T02:13:37.7136844Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7137135Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-11-23T02:13:37.7137141Z 2022-11-23T02:13:37.7137407Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7137504Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7137509Z 2022-11-23T02:13:37.7137604Z OK (skipped=1) 2022-11-23T02:13:37.7137610Z 2022-11-23T02:13:37.7137713Z Generating XML reports... 2022-11-23T02:13:37.7138144Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015019.xml 2022-11-23T02:13:37.7138453Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7138822Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7138991Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7139367Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7139544Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7139549Z 2022-11-23T02:13:37.7139646Z Running tests... 2022-11-23T02:13:37.7139910Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7140193Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:13:37.7140200Z 2022-11-23T02:13:37.7140466Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7140562Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7140568Z 2022-11-23T02:13:37.7140666Z OK (skipped=1) 2022-11-23T02:13:37.7140672Z 2022-11-23T02:13:37.7140783Z Generating XML reports... 2022-11-23T02:13:37.7141228Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015023.xml 2022-11-23T02:13:37.7141539Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7141906Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7142072Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7142453Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7142627Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7142633Z 2022-11-23T02:13:37.7142730Z Running tests... 2022-11-23T02:13:37.7142993Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7143286Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-11-23T02:13:37.7143347Z 2022-11-23T02:13:37.7143614Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7143702Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7143708Z 2022-11-23T02:13:37.7143802Z OK (skipped=1) 2022-11-23T02:13:37.7143808Z 2022-11-23T02:13:37.7143922Z Generating XML reports... 2022-11-23T02:13:37.7144357Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015027.xml 2022-11-23T02:13:37.7144667Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7145035Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7145197Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7145622Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7145802Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7145808Z 2022-11-23T02:13:37.7145910Z Running tests... 2022-11-23T02:13:37.7146177Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7146452Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:13:37.7146458Z 2022-11-23T02:13:37.7146719Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7146815Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7146820Z 2022-11-23T02:13:37.7146913Z OK (skipped=1) 2022-11-23T02:13:37.7146918Z 2022-11-23T02:13:37.7147030Z Generating XML reports... 2022-11-23T02:13:37.7147473Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015031.xml 2022-11-23T02:13:37.7147790Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7148163Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7148322Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7148703Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7148877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7148883Z 2022-11-23T02:13:37.7148970Z Running tests... 2022-11-23T02:13:37.7149232Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7149515Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-11-23T02:13:37.7149521Z 2022-11-23T02:13:37.7149789Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7149886Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7149892Z 2022-11-23T02:13:37.7149989Z OK (skipped=1) 2022-11-23T02:13:37.7149994Z 2022-11-23T02:13:37.7150107Z Generating XML reports... 2022-11-23T02:13:37.7150540Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015035.xml 2022-11-23T02:13:37.7150849Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7151219Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7151381Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7151759Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7151932Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7151996Z 2022-11-23T02:13:37.7152096Z Running tests... 2022-11-23T02:13:37.7152362Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7152664Z test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18792 2022-11-23T02:13:37.7152868Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18793 2022-11-23T02:13:37.7153121Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7153490Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7153652Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7154029Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7154257Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7154483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7154873Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7155238Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7155399Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7155779Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7155954Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7156176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7156582Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7156798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7157011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7157233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:13:37.7157452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:13:37.7157843Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.7158232Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.7158507Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:13:37.7158788Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:13:37.7158879Z ok (5.643s) 2022-11-23T02:13:37.7158885Z 2022-11-23T02:13:37.7159149Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7159245Z Ran 1 test in 5.643s 2022-11-23T02:13:37.7159251Z 2022-11-23T02:13:37.7159332Z OK 2022-11-23T02:13:37.7159339Z 2022-11-23T02:13:37.7159450Z Generating XML reports... 2022-11-23T02:13:37.7159886Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015039.xml 2022-11-23T02:13:37.7160195Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7160564Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7160717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7161165Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7161342Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7161347Z 2022-11-23T02:13:37.7161445Z Running tests... 2022-11-23T02:13:37.7161707Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7162139Z test_backend_full_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.001s) 2022-11-23T02:13:37.7162145Z 2022-11-23T02:13:37.7162406Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7162504Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7162510Z 2022-11-23T02:13:37.7162605Z OK (skipped=1) 2022-11-23T02:13:37.7162610Z 2022-11-23T02:13:37.7162724Z Generating XML reports... 2022-11-23T02:13:37.7163224Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015049.xml 2022-11-23T02:13:37.7163542Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7163910Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7164071Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7164450Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7164625Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7164631Z 2022-11-23T02:13:37.7164729Z Running tests... 2022-11-23T02:13:37.7164990Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7165406Z test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.001s) 2022-11-23T02:13:37.7165412Z 2022-11-23T02:13:37.7165681Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7165778Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7165784Z 2022-11-23T02:13:37.7165881Z OK (skipped=1) 2022-11-23T02:13:37.7165887Z 2022-11-23T02:13:37.7166000Z Generating XML reports... 2022-11-23T02:13:37.7166426Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015053.xml 2022-11-23T02:13:37.7166734Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7167105Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7167267Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7167644Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7167929Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7167938Z 2022-11-23T02:13:37.7168036Z Running tests... 2022-11-23T02:13:37.7168302Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7168588Z test_barrier (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19144 2022-11-23T02:13:37.7168792Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19145 2022-11-23T02:13:37.7169041Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7169411Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7169573Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7169954Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7170206Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7170431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7170828Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7171193Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7171356Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7171737Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7171912Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7172135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7172571Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7172793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7173003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7173093Z ok (5.712s) 2022-11-23T02:13:37.7173099Z 2022-11-23T02:13:37.7173367Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7173463Z Ran 1 test in 5.712s 2022-11-23T02:13:37.7173469Z 2022-11-23T02:13:37.7173552Z OK 2022-11-23T02:13:37.7173557Z 2022-11-23T02:13:37.7173668Z Generating XML reports... 2022-11-23T02:13:37.7174104Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015057.xml 2022-11-23T02:13:37.7174413Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7174794Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7174954Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7175332Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7175509Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7175515Z 2022-11-23T02:13:37.7175613Z Running tests... 2022-11-23T02:13:37.7175876Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7176168Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19351 2022-11-23T02:13:37.7176374Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19352 2022-11-23T02:13:37.7176632Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7177002Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7177163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7177540Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7177706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7177928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7178295Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7178455Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7178839Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7179085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7179308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7179701Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7180158Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7180409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7180660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7180767Z ok (6.313s) 2022-11-23T02:13:37.7180774Z 2022-11-23T02:13:37.7181093Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7181211Z Ran 1 test in 6.314s 2022-11-23T02:13:37.7181276Z 2022-11-23T02:13:37.7181376Z OK 2022-11-23T02:13:37.7181383Z 2022-11-23T02:13:37.7181522Z Generating XML reports... 2022-11-23T02:13:37.7182055Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015107.xml 2022-11-23T02:13:37.7182428Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7182873Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7183069Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7183521Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7183737Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7183745Z 2022-11-23T02:13:37.7183862Z Running tests... 2022-11-23T02:13:37.7184177Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7184536Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19558 2022-11-23T02:13:37.7184779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19559 2022-11-23T02:13:37.7185081Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7185530Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7185721Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7186177Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7186386Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7186658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7187096Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7187286Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7187742Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7187951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7188215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7188689Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7189157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7189496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7189744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7190009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:13:37.7190275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:13:37.7190753Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.7191223Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.7191333Z ok (6.152s) 2022-11-23T02:13:37.7191343Z 2022-11-23T02:13:37.7191652Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7191767Z Ran 1 test in 6.152s 2022-11-23T02:13:37.7191833Z 2022-11-23T02:13:37.7191937Z OK 2022-11-23T02:13:37.7191944Z 2022-11-23T02:13:37.7192077Z Generating XML reports... 2022-11-23T02:13:37.7192609Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015117.xml 2022-11-23T02:13:37.7192990Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7193433Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7193626Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7194087Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7194291Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7194298Z 2022-11-23T02:13:37.7194416Z Running tests... 2022-11-23T02:13:37.7194740Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7195107Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19771 2022-11-23T02:13:37.7195356Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19772 2022-11-23T02:13:37.7195666Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7196094Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7196254Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7196633Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7196811Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7197038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7197437Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7197803Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7197954Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7198332Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7198507Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7198726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7199119Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7199392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7199602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7199751Z skip: Skipped due to small world size. (5.009s) 2022-11-23T02:13:37.7199758Z 2022-11-23T02:13:37.7200027Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7200126Z Ran 1 test in 5.010s 2022-11-23T02:13:37.7200132Z 2022-11-23T02:13:37.7200229Z OK (skipped=1) 2022-11-23T02:13:37.7200235Z 2022-11-23T02:13:37.7200347Z Generating XML reports... 2022-11-23T02:13:37.7200787Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015128.xml 2022-11-23T02:13:37.7201098Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7201516Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7201685Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7202070Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7202245Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7202251Z 2022-11-23T02:13:37.7202347Z Running tests... 2022-11-23T02:13:37.7202613Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7202907Z test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19978 2022-11-23T02:13:37.7203112Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19979 2022-11-23T02:13:37.7203362Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7203728Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7203889Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7204269Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7204445Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7204671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7205037Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7205199Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7205573Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7205749Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7205972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7206367Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7206756Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7206966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7207181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7207328Z skip: Skipped due to small world size. (4.807s) 2022-11-23T02:13:37.7207334Z 2022-11-23T02:13:37.7207597Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7207738Z Ran 1 test in 4.808s 2022-11-23T02:13:37.7207744Z 2022-11-23T02:13:37.7207907Z OK (skipped=1) 2022-11-23T02:13:37.7207915Z 2022-11-23T02:13:37.7208029Z Generating XML reports... 2022-11-23T02:13:37.7208477Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015137.xml 2022-11-23T02:13:37.7208788Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7209158Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7209310Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7209691Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7209863Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7209869Z 2022-11-23T02:13:37.7209965Z Running tests... 2022-11-23T02:13:37.7210283Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7210592Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20185 2022-11-23T02:13:37.7210794Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20186 2022-11-23T02:13:37.7211045Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7211417Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7211577Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7211958Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7212132Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7212356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7212729Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7212889Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7213267Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7213444Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7213664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7214056Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7214444Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7214664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7214873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7215020Z skip: Skipped due to small world size. (5.012s) 2022-11-23T02:13:37.7215026Z 2022-11-23T02:13:37.7215289Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7215377Z Ran 1 test in 5.012s 2022-11-23T02:13:37.7215384Z 2022-11-23T02:13:37.7215481Z OK (skipped=1) 2022-11-23T02:13:37.7215487Z 2022-11-23T02:13:37.7215599Z Generating XML reports... 2022-11-23T02:13:37.7216037Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015145.xml 2022-11-23T02:13:37.7216344Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7216712Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7216939Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7217320Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7217497Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7217503Z 2022-11-23T02:13:37.7217599Z Running tests... 2022-11-23T02:13:37.7217864Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7218172Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20392 2022-11-23T02:13:37.7218377Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20393 2022-11-23T02:13:37.7218629Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7219047Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7219213Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7219597Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7219771Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7219993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7220360Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7220521Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7220902Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7221073Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7221299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7221689Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7222077Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7222287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7222498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7222717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:13:37.7222938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:13:37.7223332Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.7223723Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.7223813Z ok (6.211s) 2022-11-23T02:13:37.7223819Z 2022-11-23T02:13:37.7224081Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7224178Z Ran 1 test in 6.212s 2022-11-23T02:13:37.7224184Z 2022-11-23T02:13:37.7224265Z OK 2022-11-23T02:13:37.7224270Z 2022-11-23T02:13:37.7224381Z Generating XML reports... 2022-11-23T02:13:37.7224818Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015155.xml 2022-11-23T02:13:37.7225125Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7225498Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7225717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7226102Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7226280Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7226288Z 2022-11-23T02:13:37.7226384Z Running tests... 2022-11-23T02:13:37.7226640Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7227806Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Requires file:// initialization method. Both tcp:// and env:// rely on the TCP store for which reinitialization has proven racy. (0.002s) 2022-11-23T02:13:37.7227831Z 2022-11-23T02:13:37.7228116Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7228221Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7228226Z 2022-11-23T02:13:37.7228324Z OK (skipped=1) 2022-11-23T02:13:37.7228404Z 2022-11-23T02:13:37.7228530Z Generating XML reports... 2022-11-23T02:13:37.7228995Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015205.xml 2022-11-23T02:13:37.7229315Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7229690Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7229862Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7230260Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7230445Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7230451Z 2022-11-23T02:13:37.7230557Z Running tests... 2022-11-23T02:13:37.7230838Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7231157Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20671 2022-11-23T02:13:37.7231368Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20672 2022-11-23T02:13:37.7231635Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7232011Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7232182Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7232589Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7232767Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7233010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7233455Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7233820Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7233990Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7234395Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7234577Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7234799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7235208Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7235443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7235748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7235911Z skip: Skipped due to small world size. (4.906s) 2022-11-23T02:13:37.7235917Z 2022-11-23T02:13:37.7236205Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7236310Z Ran 1 test in 4.907s 2022-11-23T02:13:37.7236316Z 2022-11-23T02:13:37.7236419Z OK (skipped=1) 2022-11-23T02:13:37.7236424Z 2022-11-23T02:13:37.7236546Z Generating XML reports... 2022-11-23T02:13:37.7237017Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015209.xml 2022-11-23T02:13:37.7237333Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7237717Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7237936Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7238342Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7238524Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7238530Z 2022-11-23T02:13:37.7238626Z Running tests... 2022-11-23T02:13:37.7238893Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7239196Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20878 2022-11-23T02:13:37.7239397Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20879 2022-11-23T02:13:37.7239648Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7240019Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7240177Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7240553Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7240734Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7240965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7241335Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7241496Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7241877Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7242053Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7242276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7242668Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7243059Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7243271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7243479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7243571Z ok (5.414s) 2022-11-23T02:13:37.7243577Z 2022-11-23T02:13:37.7243840Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7243939Z Ran 1 test in 5.414s 2022-11-23T02:13:37.7243945Z 2022-11-23T02:13:37.7244027Z OK 2022-11-23T02:13:37.7244032Z 2022-11-23T02:13:37.7244199Z Generating XML reports... 2022-11-23T02:13:37.7244644Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015218.xml 2022-11-23T02:13:37.7244956Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7245324Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7245476Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7245886Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7246060Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7246066Z 2022-11-23T02:13:37.7246161Z Running tests... 2022-11-23T02:13:37.7246430Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7246792Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21085 2022-11-23T02:13:37.7247004Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21086 2022-11-23T02:13:37.7247250Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7247619Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7247824Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7248203Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7248379Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7248601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7248975Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7249139Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7249516Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7249690Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7249913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7250306Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7250696Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7250910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7251126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7251218Z ok (5.009s) 2022-11-23T02:13:37.7251224Z 2022-11-23T02:13:37.7251480Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7251581Z Ran 1 test in 5.009s 2022-11-23T02:13:37.7251587Z 2022-11-23T02:13:37.7251669Z OK 2022-11-23T02:13:37.7251674Z 2022-11-23T02:13:37.7251789Z Generating XML reports... 2022-11-23T02:13:37.7252225Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015227.xml 2022-11-23T02:13:37.7252533Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7252898Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7253059Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7253516Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7253693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7253699Z 2022-11-23T02:13:37.7253797Z Running tests... 2022-11-23T02:13:37.7254062Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7254320Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-11-23T02:13:37.7254326Z 2022-11-23T02:13:37.7254588Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7254686Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7254692Z 2022-11-23T02:13:37.7254788Z OK (skipped=1) 2022-11-23T02:13:37.7254794Z 2022-11-23T02:13:37.7254905Z Generating XML reports... 2022-11-23T02:13:37.7255342Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015237.xml 2022-11-23T02:13:37.7255709Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7256085Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7256247Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7256627Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7256794Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7256813Z 2022-11-23T02:13:37.7256900Z Running tests... 2022-11-23T02:13:37.7257165Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7257404Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s) 2022-11-23T02:13:37.7257410Z 2022-11-23T02:13:37.7257676Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7257780Z Ran 1 test in 0.003s 2022-11-23T02:13:37.7257786Z 2022-11-23T02:13:37.7257885Z OK (skipped=1) 2022-11-23T02:13:37.7257890Z 2022-11-23T02:13:37.7258005Z Generating XML reports... 2022-11-23T02:13:37.7258442Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015241.xml 2022-11-23T02:13:37.7258750Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7259121Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7259282Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7259661Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7259840Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7259855Z 2022-11-23T02:13:37.7259953Z Running tests... 2022-11-23T02:13:37.7260219Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7260468Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-11-23T02:13:37.7260474Z 2022-11-23T02:13:37.7260737Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7260836Z Ran 1 test in 0.003s 2022-11-23T02:13:37.7260842Z 2022-11-23T02:13:37.7260939Z OK (skipped=1) 2022-11-23T02:13:37.7260944Z 2022-11-23T02:13:37.7261056Z Generating XML reports... 2022-11-23T02:13:37.7261503Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015245.xml 2022-11-23T02:13:37.7261815Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7262176Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7262413Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7262793Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7262970Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7262975Z 2022-11-23T02:13:37.7263074Z Running tests... 2022-11-23T02:13:37.7263338Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7263580Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-11-23T02:13:37.7263586Z 2022-11-23T02:13:37.7263847Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7263947Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7263952Z 2022-11-23T02:13:37.7264050Z OK (skipped=1) 2022-11-23T02:13:37.7264056Z 2022-11-23T02:13:37.7264227Z Generating XML reports... 2022-11-23T02:13:37.7264672Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015249.xml 2022-11-23T02:13:37.7264987Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7265358Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7265519Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7265900Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7266078Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7266084Z 2022-11-23T02:13:37.7266182Z Running tests... 2022-11-23T02:13:37.7266446Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7266699Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-11-23T02:13:37.7266709Z 2022-11-23T02:13:37.7266974Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7267072Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7267078Z 2022-11-23T02:13:37.7267176Z OK (skipped=1) 2022-11-23T02:13:37.7267182Z 2022-11-23T02:13:37.7267285Z Generating XML reports... 2022-11-23T02:13:37.7267718Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015253.xml 2022-11-23T02:13:37.7268025Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7268396Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7268560Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7268941Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7269121Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7269127Z 2022-11-23T02:13:37.7269225Z Running tests... 2022-11-23T02:13:37.7269490Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7269746Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-11-23T02:13:37.7269754Z 2022-11-23T02:13:37.7270016Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7270115Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7270121Z 2022-11-23T02:13:37.7270216Z OK (skipped=1) 2022-11-23T02:13:37.7270222Z 2022-11-23T02:13:37.7270334Z Generating XML reports... 2022-11-23T02:13:37.7270768Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015257.xml 2022-11-23T02:13:37.7271139Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7271510Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7271671Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7272054Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7272232Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7272238Z 2022-11-23T02:13:37.7272335Z Running tests... 2022-11-23T02:13:37.7272603Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7272839Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-11-23T02:13:37.7272854Z 2022-11-23T02:13:37.7273107Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7273257Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7273263Z 2022-11-23T02:13:37.7273364Z OK (skipped=1) 2022-11-23T02:13:37.7273370Z 2022-11-23T02:13:37.7273483Z Generating XML reports... 2022-11-23T02:13:37.7273923Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015301.xml 2022-11-23T02:13:37.7274229Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7274596Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7274757Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7275140Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7275317Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7275322Z 2022-11-23T02:13:37.7275429Z Running tests... 2022-11-23T02:13:37.7275702Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7275949Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-11-23T02:13:37.7275955Z 2022-11-23T02:13:37.7276221Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7276322Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7276327Z 2022-11-23T02:13:37.7276422Z OK (skipped=1) 2022-11-23T02:13:37.7276428Z 2022-11-23T02:13:37.7276542Z Generating XML reports... 2022-11-23T02:13:37.7276980Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015305.xml 2022-11-23T02:13:37.7277290Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7277657Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7277826Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7278207Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7278374Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7278390Z 2022-11-23T02:13:37.7278478Z Running tests... 2022-11-23T02:13:37.7278741Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7279030Z test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21820 2022-11-23T02:13:37.7279234Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21821 2022-11-23T02:13:37.7279482Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7279853Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7280076Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7280459Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7280634Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7280860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7281257Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7281623Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7281786Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7282256Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7282445Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7282668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7283066Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7283278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7283611Z STAGE:2022-11-23 01:53:12 21821:21821 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7283824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7284151Z STAGE:2022-11-23 01:53:12 21820:21820 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7284492Z STAGE:2022-11-23 01:53:12 21820:21820 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7284832Z STAGE:2022-11-23 01:53:12 21820:21820 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7285166Z STAGE:2022-11-23 01:53:12 21821:21821 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7285517Z STAGE:2022-11-23 01:53:12 21821:21821 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7285844Z STAGE:2022-11-23 01:53:12 21820:21820 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7286165Z STAGE:2022-11-23 01:53:12 21821:21821 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7286497Z STAGE:2022-11-23 01:53:12 21820:21820 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7286827Z STAGE:2022-11-23 01:53:12 21821:21821 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7287177Z STAGE:2022-11-23 01:53:12 21820:21820 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7287529Z STAGE:2022-11-23 01:53:12 21821:21821 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7287975Z STAGE:2022-11-23 01:53:12 21821:21821 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7288304Z STAGE:2022-11-23 01:53:12 21820:21820 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7288633Z STAGE:2022-11-23 01:53:12 21820:21820 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7288979Z STAGE:2022-11-23 01:53:12 21820:21820 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7289309Z STAGE:2022-11-23 01:53:12 21821:21821 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7289653Z STAGE:2022-11-23 01:53:12 21821:21821 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7290056Z STAGE:2022-11-23 01:53:12 21820:21820 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7290379Z STAGE:2022-11-23 01:53:12 21821:21821 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7290713Z STAGE:2022-11-23 01:53:12 21821:21821 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7291057Z STAGE:2022-11-23 01:53:12 21821:21821 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7291388Z STAGE:2022-11-23 01:53:12 21820:21820 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7291736Z STAGE:2022-11-23 01:53:12 21820:21820 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7292062Z STAGE:2022-11-23 01:53:12 21821:21821 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7292385Z STAGE:2022-11-23 01:53:12 21820:21820 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7292783Z STAGE:2022-11-23 01:53:12 21821:21821 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7293136Z STAGE:2022-11-23 01:53:12 21821:21821 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7293459Z STAGE:2022-11-23 01:53:12 21820:21820 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7293800Z STAGE:2022-11-23 01:53:12 21820:21820 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7294123Z STAGE:2022-11-23 01:53:12 21821:21821 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7294446Z STAGE:2022-11-23 01:53:12 21820:21820 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7294777Z STAGE:2022-11-23 01:53:12 21821:21821 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7295122Z STAGE:2022-11-23 01:53:12 21821:21821 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7295461Z STAGE:2022-11-23 01:53:12 21820:21820 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7295805Z STAGE:2022-11-23 01:53:12 21820:21820 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7296127Z STAGE:2022-11-23 01:53:12 21821:21821 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7296450Z STAGE:2022-11-23 01:53:12 21820:21820 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7296780Z STAGE:2022-11-23 01:53:12 21821:21821 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7297124Z STAGE:2022-11-23 01:53:12 21821:21821 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7297454Z STAGE:2022-11-23 01:53:12 21820:21820 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7297801Z STAGE:2022-11-23 01:53:12 21820:21820 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7298126Z STAGE:2022-11-23 01:53:12 21821:21821 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7298449Z STAGE:2022-11-23 01:53:12 21820:21820 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7298782Z STAGE:2022-11-23 01:53:12 21821:21821 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7299126Z STAGE:2022-11-23 01:53:12 21821:21821 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7299458Z STAGE:2022-11-23 01:53:12 21820:21820 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7299802Z STAGE:2022-11-23 01:53:12 21820:21820 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7300127Z STAGE:2022-11-23 01:53:12 21821:21821 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7300516Z STAGE:2022-11-23 01:53:12 21820:21820 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7300848Z STAGE:2022-11-23 01:53:12 21820:21820 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7301192Z STAGE:2022-11-23 01:53:12 21820:21820 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7301511Z STAGE:2022-11-23 01:53:12 21821:21821 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7301853Z STAGE:2022-11-23 01:53:12 21821:21821 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7302177Z STAGE:2022-11-23 01:53:12 21820:21820 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7302500Z STAGE:2022-11-23 01:53:12 21821:21821 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7302830Z STAGE:2022-11-23 01:53:12 21821:21821 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7303224Z STAGE:2022-11-23 01:53:12 21821:21821 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7303563Z STAGE:2022-11-23 01:53:12 21820:21820 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7303905Z STAGE:2022-11-23 01:53:12 21820:21820 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7304228Z STAGE:2022-11-23 01:53:12 21821:21821 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7304549Z STAGE:2022-11-23 01:53:12 21820:21820 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7304881Z STAGE:2022-11-23 01:53:12 21820:21820 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7305226Z STAGE:2022-11-23 01:53:12 21820:21820 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7305562Z STAGE:2022-11-23 01:53:12 21821:21821 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7305910Z STAGE:2022-11-23 01:53:12 21821:21821 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7306237Z STAGE:2022-11-23 01:53:12 21820:21820 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7306559Z STAGE:2022-11-23 01:53:12 21821:21821 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7306888Z STAGE:2022-11-23 01:53:12 21820:21820 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7307232Z STAGE:2022-11-23 01:53:12 21820:21820 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7307562Z STAGE:2022-11-23 01:53:12 21821:21821 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7307907Z STAGE:2022-11-23 01:53:12 21821:21821 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7307998Z ok (5.014s) 2022-11-23T02:13:37.7308007Z 2022-11-23T02:13:37.7308275Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7308375Z Ran 1 test in 5.015s 2022-11-23T02:13:37.7308381Z 2022-11-23T02:13:37.7308453Z OK 2022-11-23T02:13:37.7308468Z 2022-11-23T02:13:37.7308571Z Generating XML reports... 2022-11-23T02:13:37.7309008Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015309.xml 2022-11-23T02:13:37.7309317Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7309686Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7309850Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7310231Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7310412Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7310472Z 2022-11-23T02:13:37.7310571Z Running tests... 2022-11-23T02:13:37.7310840Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7311741Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.584s) 2022-11-23T02:13:37.7311748Z 2022-11-23T02:13:37.7312011Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7312110Z Ran 1 test in 0.584s 2022-11-23T02:13:37.7312116Z 2022-11-23T02:13:37.7312214Z OK (skipped=1) 2022-11-23T02:13:37.7312220Z 2022-11-23T02:13:37.7312334Z Generating XML reports... 2022-11-23T02:13:37.7312821Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015318.xml 2022-11-23T02:13:37.7313142Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7313512Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7313675Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7314053Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7314234Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7314241Z 2022-11-23T02:13:37.7314339Z Running tests... 2022-11-23T02:13:37.7314604Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7314901Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22099 2022-11-23T02:13:37.7315115Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22100 2022-11-23T02:13:37.7315366Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7315723Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7315884Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7316262Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7316440Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7316664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7319250Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7319617Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7319779Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7320161Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7320340Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7320565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7320960Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7321175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7321387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7321682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:13:37.7321906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:13:37.7322298Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.7322692Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.7323019Z STAGE:2022-11-23 01:53:25 22099:22099 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7323346Z STAGE:2022-11-23 01:53:25 22100:22100 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7323678Z STAGE:2022-11-23 01:53:25 22099:22099 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7324049Z STAGE:2022-11-23 01:53:25 22100:22100 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7324407Z STAGE:2022-11-23 01:53:25 22099:22099 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7324750Z STAGE:2022-11-23 01:53:25 22100:22100 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7325080Z STAGE:2022-11-23 01:53:25 22099:22099 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7325391Z STAGE:2022-11-23 01:53:25 22100:22100 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7325718Z STAGE:2022-11-23 01:53:25 22100:22100 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7326048Z STAGE:2022-11-23 01:53:25 22099:22099 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7326391Z STAGE:2022-11-23 01:53:25 22100:22100 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7326743Z STAGE:2022-11-23 01:53:25 22099:22099 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7327066Z STAGE:2022-11-23 01:53:25 22100:22100 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7327391Z STAGE:2022-11-23 01:53:25 22099:22099 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7327760Z STAGE:2022-11-23 01:53:25 22099:22099 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7328105Z STAGE:2022-11-23 01:53:25 22099:22099 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7328436Z STAGE:2022-11-23 01:53:25 22100:22100 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7328778Z STAGE:2022-11-23 01:53:25 22100:22100 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7329103Z STAGE:2022-11-23 01:53:25 22099:22099 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7329436Z STAGE:2022-11-23 01:53:25 22100:22100 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7329764Z STAGE:2022-11-23 01:53:25 22100:22100 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7330106Z STAGE:2022-11-23 01:53:25 22100:22100 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7330437Z STAGE:2022-11-23 01:53:25 22099:22099 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7330778Z STAGE:2022-11-23 01:53:25 22099:22099 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7331100Z STAGE:2022-11-23 01:53:25 22100:22100 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7331424Z STAGE:2022-11-23 01:53:25 22099:22099 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7331756Z STAGE:2022-11-23 01:53:25 22100:22100 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7332166Z STAGE:2022-11-23 01:53:25 22100:22100 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7332498Z STAGE:2022-11-23 01:53:25 22099:22099 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7332842Z STAGE:2022-11-23 01:53:25 22099:22099 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7333164Z STAGE:2022-11-23 01:53:25 22100:22100 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7333477Z STAGE:2022-11-23 01:53:25 22099:22099 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7333803Z STAGE:2022-11-23 01:53:25 22100:22100 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7334143Z STAGE:2022-11-23 01:53:25 22100:22100 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7334522Z STAGE:2022-11-23 01:53:25 22099:22099 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7334877Z STAGE:2022-11-23 01:53:25 22099:22099 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7335199Z STAGE:2022-11-23 01:53:25 22100:22100 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7335517Z STAGE:2022-11-23 01:53:25 22099:22099 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7335853Z STAGE:2022-11-23 01:53:25 22100:22100 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7336196Z STAGE:2022-11-23 01:53:25 22100:22100 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7336525Z STAGE:2022-11-23 01:53:25 22099:22099 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7336869Z STAGE:2022-11-23 01:53:25 22099:22099 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7337409Z STAGE:2022-11-23 01:53:25 22100:22100 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:53:25 22099:22099 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7337416Z 2022-11-23T02:13:37.7337749Z STAGE:2022-11-23 01:53:25 22099:22099 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7338089Z STAGE:2022-11-23 01:53:25 22099:22099 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7338418Z STAGE:2022-11-23 01:53:25 22100:22100 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7338764Z STAGE:2022-11-23 01:53:25 22100:22100 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7339089Z STAGE:2022-11-23 01:53:25 22099:22099 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7339414Z STAGE:2022-11-23 01:53:25 22100:22100 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7339747Z STAGE:2022-11-23 01:53:25 22099:22099 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7340091Z STAGE:2022-11-23 01:53:25 22099:22099 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7340420Z STAGE:2022-11-23 01:53:25 22100:22100 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7340763Z STAGE:2022-11-23 01:53:25 22100:22100 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7341087Z STAGE:2022-11-23 01:53:25 22099:22099 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7341411Z STAGE:2022-11-23 01:53:25 22100:22100 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7341742Z STAGE:2022-11-23 01:53:25 22099:22099 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7342085Z STAGE:2022-11-23 01:53:25 22099:22099 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7342472Z STAGE:2022-11-23 01:53:25 22100:22100 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7342809Z STAGE:2022-11-23 01:53:25 22100:22100 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7343131Z STAGE:2022-11-23 01:53:25 22099:22099 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7343455Z STAGE:2022-11-23 01:53:25 22100:22100 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7343786Z STAGE:2022-11-23 01:53:25 22099:22099 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7344130Z STAGE:2022-11-23 01:53:25 22099:22099 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7344459Z STAGE:2022-11-23 01:53:25 22100:22100 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7344849Z STAGE:2022-11-23 01:53:25 22100:22100 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7345184Z STAGE:2022-11-23 01:53:25 22099:22099 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7345505Z STAGE:2022-11-23 01:53:25 22100:22100 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7345840Z STAGE:2022-11-23 01:53:25 22099:22099 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7346182Z STAGE:2022-11-23 01:53:25 22099:22099 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7346512Z STAGE:2022-11-23 01:53:25 22100:22100 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7346853Z STAGE:2022-11-23 01:53:25 22100:22100 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7346944Z ok (5.235s) 2022-11-23T02:13:37.7346950Z 2022-11-23T02:13:37.7347216Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7347323Z Ran 1 test in 5.236s 2022-11-23T02:13:37.7347328Z 2022-11-23T02:13:37.7347409Z OK 2022-11-23T02:13:37.7347415Z 2022-11-23T02:13:37.7347526Z Generating XML reports... 2022-11-23T02:13:37.7347965Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015323.xml 2022-11-23T02:13:37.7348274Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7348643Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7348804Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7349172Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7349351Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7349367Z 2022-11-23T02:13:37.7349455Z Running tests... 2022-11-23T02:13:37.7349728Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7350024Z test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22318 2022-11-23T02:13:37.7350229Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22319 2022-11-23T02:13:37.7350479Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7350846Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7351009Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7351390Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7351564Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7351853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7352252Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7352620Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7352782Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7353157Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7353333Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7353557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7353997Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7354218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7354429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7354576Z skip: Skipped due to small world size. (4.812s) 2022-11-23T02:13:37.7354582Z 2022-11-23T02:13:37.7354848Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7354938Z Ran 1 test in 4.812s 2022-11-23T02:13:37.7354952Z 2022-11-23T02:13:37.7355038Z OK (skipped=1) 2022-11-23T02:13:37.7355044Z 2022-11-23T02:13:37.7355155Z Generating XML reports... 2022-11-23T02:13:37.7355591Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015332.xml 2022-11-23T02:13:37.7355902Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7356275Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7356439Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7356820Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7356998Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7357003Z 2022-11-23T02:13:37.7357102Z Running tests... 2022-11-23T02:13:37.7357368Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7357663Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22525 2022-11-23T02:13:37.7357866Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22526 2022-11-23T02:13:37.7358119Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7358491Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7358655Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7359032Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7359210Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7359431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7359798Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7359959Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7360338Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7360575Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7360790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7361188Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7361572Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7361784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7361999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7362821Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1402: UserWarning: torch.distributed.broadcast_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2022-11-23T02:13:37.7362930Z warnings.warn( 2022-11-23T02:13:37.7363705Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1402: UserWarning: torch.distributed.broadcast_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2022-11-23T02:13:37.7363805Z warnings.warn( 2022-11-23T02:13:37.7363896Z ok (5.306s) 2022-11-23T02:13:37.7363902Z 2022-11-23T02:13:37.7364166Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7364266Z Ran 1 test in 5.307s 2022-11-23T02:13:37.7364272Z 2022-11-23T02:13:37.7364353Z OK 2022-11-23T02:13:37.7364359Z 2022-11-23T02:13:37.7364471Z Generating XML reports... 2022-11-23T02:13:37.7364912Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015341.xml 2022-11-23T02:13:37.7365233Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7365600Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7365764Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7366142Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7366319Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7366325Z 2022-11-23T02:13:37.7366422Z Running tests... 2022-11-23T02:13:37.7366684Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7367590Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.578s) 2022-11-23T02:13:37.7367601Z 2022-11-23T02:13:37.7367897Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7367997Z Ran 1 test in 0.578s 2022-11-23T02:13:37.7368003Z 2022-11-23T02:13:37.7368101Z OK (skipped=1) 2022-11-23T02:13:37.7368106Z 2022-11-23T02:13:37.7368220Z Generating XML reports... 2022-11-23T02:13:37.7368647Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015351.xml 2022-11-23T02:13:37.7368956Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7369325Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7369487Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7369957Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7370131Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7370137Z 2022-11-23T02:13:37.7370234Z Running tests... 2022-11-23T02:13:37.7370498Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7371012Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.001s) 2022-11-23T02:13:37.7371019Z 2022-11-23T02:13:37.7371279Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7371378Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7371384Z 2022-11-23T02:13:37.7371480Z OK (skipped=1) 2022-11-23T02:13:37.7371485Z 2022-11-23T02:13:37.7371599Z Generating XML reports... 2022-11-23T02:13:37.7372093Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015355.xml 2022-11-23T02:13:37.7372414Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7372784Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7372944Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7373324Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7373499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7373505Z 2022-11-23T02:13:37.7373605Z Running tests... 2022-11-23T02:13:37.7373871Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7374384Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.001s) 2022-11-23T02:13:37.7374394Z 2022-11-23T02:13:37.7374656Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7374756Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7374762Z 2022-11-23T02:13:37.7374857Z OK (skipped=1) 2022-11-23T02:13:37.7374863Z 2022-11-23T02:13:37.7374966Z Generating XML reports... 2022-11-23T02:13:37.7375401Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015359.xml 2022-11-23T02:13:37.7375713Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7376080Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7376245Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7376626Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7376804Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7376810Z 2022-11-23T02:13:37.7376906Z Running tests... 2022-11-23T02:13:37.7377172Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7377476Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22930 2022-11-23T02:13:37.7377680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22931 2022-11-23T02:13:37.7377932Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7378301Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7378463Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7378903Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7379079Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7379303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7379666Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7379828Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7380204Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7380381Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7380601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7381029Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7381425Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7381640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7381852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7382085Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpitjjq8u3 2022-11-23T02:13:37.7382334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpitjjq8u3/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7382569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsiecne_t 2022-11-23T02:13:37.7382821Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsiecne_t/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7383042Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7383257Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7383350Z ok (7.611s) 2022-11-23T02:13:37.7383356Z 2022-11-23T02:13:37.7383621Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7383719Z Ran 1 test in 7.611s 2022-11-23T02:13:37.7383725Z 2022-11-23T02:13:37.7383806Z OK 2022-11-23T02:13:37.7383811Z 2022-11-23T02:13:37.7383926Z Generating XML reports... 2022-11-23T02:13:37.7384361Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015403.xml 2022-11-23T02:13:37.7384667Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7385031Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7385198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7385587Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7385763Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7385769Z 2022-11-23T02:13:37.7385865Z Running tests... 2022-11-23T02:13:37.7386119Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7386432Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23147 2022-11-23T02:13:37.7386633Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23148 2022-11-23T02:13:37.7386883Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7387253Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7387547Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7387930Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7388108Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7388332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7388701Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7388860Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7389236Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7389413Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7389695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7390099Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7390493Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7390705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7390916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7391144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbnas_8ps 2022-11-23T02:13:37.7391390Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbnas_8ps/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7391621Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpijtt7iiw 2022-11-23T02:13:37.7391876Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpijtt7iiw/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7392092Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7392292Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7392503Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7392718Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7392808Z ok (7.319s) 2022-11-23T02:13:37.7392815Z 2022-11-23T02:13:37.7393083Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7393182Z Ran 1 test in 7.319s 2022-11-23T02:13:37.7393188Z 2022-11-23T02:13:37.7393270Z OK 2022-11-23T02:13:37.7393276Z 2022-11-23T02:13:37.7393389Z Generating XML reports... 2022-11-23T02:13:37.7393828Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015415.xml 2022-11-23T02:13:37.7394142Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7394510Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7394671Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7395049Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7395224Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7395231Z 2022-11-23T02:13:37.7395330Z Running tests... 2022-11-23T02:13:37.7395598Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7396511Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78641 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.584s) 2022-11-23T02:13:37.7396588Z 2022-11-23T02:13:37.7396856Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7396954Z Ran 1 test in 0.584s 2022-11-23T02:13:37.7396960Z 2022-11-23T02:13:37.7397055Z OK (skipped=1) 2022-11-23T02:13:37.7397060Z 2022-11-23T02:13:37.7397171Z Generating XML reports... 2022-11-23T02:13:37.7397608Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015426.xml 2022-11-23T02:13:37.7397918Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7398288Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7398499Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7398874Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7399047Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7399052Z 2022-11-23T02:13:37.7399150Z Running tests... 2022-11-23T02:13:37.7399413Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7400348Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.578s) 2022-11-23T02:13:37.7400356Z 2022-11-23T02:13:37.7400622Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7400724Z Ran 1 test in 0.579s 2022-11-23T02:13:37.7400730Z 2022-11-23T02:13:37.7400829Z OK (skipped=1) 2022-11-23T02:13:37.7400835Z 2022-11-23T02:13:37.7400946Z Generating XML reports... 2022-11-23T02:13:37.7401380Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015431.xml 2022-11-23T02:13:37.7401689Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7402057Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7402219Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7402597Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7402774Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7402790Z 2022-11-23T02:13:37.7402886Z Running tests... 2022-11-23T02:13:37.7403150Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7403616Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.003s) 2022-11-23T02:13:37.7403623Z 2022-11-23T02:13:37.7403884Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7403984Z Ran 1 test in 0.004s 2022-11-23T02:13:37.7403990Z 2022-11-23T02:13:37.7404084Z OK (skipped=1) 2022-11-23T02:13:37.7404090Z 2022-11-23T02:13:37.7404203Z Generating XML reports... 2022-11-23T02:13:37.7404642Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015436.xml 2022-11-23T02:13:37.7404951Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7405386Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7405547Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7405915Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7406093Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7406099Z 2022-11-23T02:13:37.7406196Z Running tests... 2022-11-23T02:13:37.7406462Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7406801Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23562 2022-11-23T02:13:37.7407005Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23563 2022-11-23T02:13:37.7407304Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7407682Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7407945Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7408325Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7408500Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7408723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7409118Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7409485Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7409654Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7410031Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7410204Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7410428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7410822Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7411031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7411241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7411472Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwwy7ys3p 2022-11-23T02:13:37.7411720Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwwy7ys3p/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7411946Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9q6s8hpw 2022-11-23T02:13:37.7412189Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9q6s8hpw/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7412280Z ok (5.268s) 2022-11-23T02:13:37.7412286Z 2022-11-23T02:13:37.7412553Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7412653Z Ran 1 test in 5.269s 2022-11-23T02:13:37.7412659Z 2022-11-23T02:13:37.7412740Z OK 2022-11-23T02:13:37.7412746Z 2022-11-23T02:13:37.7412857Z Generating XML reports... 2022-11-23T02:13:37.7413291Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015440.xml 2022-11-23T02:13:37.7413599Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7413967Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7414202Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7414579Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7414754Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7414760Z 2022-11-23T02:13:37.7414859Z Running tests... 2022-11-23T02:13:37.7415123Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7415420Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23769 2022-11-23T02:13:37.7415623Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23770 2022-11-23T02:13:37.7415873Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7416295Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7416457Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7416837Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7417014Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7417227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7417589Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7417751Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7418126Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7418310Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7418530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7418923Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7419312Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7419524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7419737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7419968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2b2l6w7w 2022-11-23T02:13:37.7420210Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2b2l6w7w/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7420445Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1drenuol 2022-11-23T02:13:37.7420689Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1drenuol/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7420906Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7421123Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7421214Z ok (7.467s) 2022-11-23T02:13:37.7421220Z 2022-11-23T02:13:37.7421489Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7421588Z Ran 1 test in 7.468s 2022-11-23T02:13:37.7421593Z 2022-11-23T02:13:37.7421674Z OK 2022-11-23T02:13:37.7421680Z 2022-11-23T02:13:37.7421796Z Generating XML reports... 2022-11-23T02:13:37.7422229Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015449.xml 2022-11-23T02:13:37.7422593Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7422961Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7423124Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7423502Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7423677Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7423684Z 2022-11-23T02:13:37.7423779Z Running tests... 2022-11-23T02:13:37.7424043Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7424505Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.005s) 2022-11-23T02:13:37.7424512Z 2022-11-23T02:13:37.7424818Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7424926Z Ran 1 test in 0.005s 2022-11-23T02:13:37.7424932Z 2022-11-23T02:13:37.7425031Z OK (skipped=1) 2022-11-23T02:13:37.7425036Z 2022-11-23T02:13:37.7425148Z Generating XML reports... 2022-11-23T02:13:37.7425592Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015501.xml 2022-11-23T02:13:37.7425899Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7426265Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7426429Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7426805Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7426981Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7426994Z 2022-11-23T02:13:37.7427093Z Running tests... 2022-11-23T02:13:37.7427356Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7427812Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.004s) 2022-11-23T02:13:37.7427818Z 2022-11-23T02:13:37.7428079Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7428177Z Ran 1 test in 0.004s 2022-11-23T02:13:37.7428183Z 2022-11-23T02:13:37.7428280Z OK (skipped=1) 2022-11-23T02:13:37.7428286Z 2022-11-23T02:13:37.7428389Z Generating XML reports... 2022-11-23T02:13:37.7428825Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015505.xml 2022-11-23T02:13:37.7429131Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7429502Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7429665Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7430042Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7430217Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7430223Z 2022-11-23T02:13:37.7430320Z Running tests... 2022-11-23T02:13:37.7430581Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7430877Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24118 2022-11-23T02:13:37.7431081Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24119 2022-11-23T02:13:37.7431340Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7431764Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7431927Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7432302Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7432479Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7432704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7433068Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7433227Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7433603Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7433845Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7434072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7434461Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7434843Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7435054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7435285Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp82i9aml_ 2022-11-23T02:13:37.7435528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp82i9aml_/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7435738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7435979Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzxc0g2dt 2022-11-23T02:13:37.7436220Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzxc0g2dt/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7437145Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:13:37.7438054Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:13:37.7439278Z /opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1122.) 2022-11-23T02:13:37.7439495Z Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2022-11-23T02:13:37.7439708Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7440915Z /opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1122.) 2022-11-23T02:13:37.7441197Z Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2022-11-23T02:13:37.7441412Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7442310Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:13:37.7443270Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:13:37.7444172Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:13:37.7445064Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:13:37.7445969Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:13:37.7446860Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:13:37.7447800Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:13:37.7448694Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:13:37.7449578Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:13:37.7450535Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:13:37.7450626Z ok (5.006s) 2022-11-23T02:13:37.7450632Z 2022-11-23T02:13:37.7450896Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7450996Z Ran 1 test in 5.006s 2022-11-23T02:13:37.7451002Z 2022-11-23T02:13:37.7451082Z OK 2022-11-23T02:13:37.7451088Z 2022-11-23T02:13:37.7451198Z Generating XML reports... 2022-11-23T02:13:37.7451636Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015509.xml 2022-11-23T02:13:37.7451999Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7452377Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7452539Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7452925Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7453102Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7453108Z 2022-11-23T02:13:37.7453205Z Running tests... 2022-11-23T02:13:37.7453470Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7453875Z test_ddp_device (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.005s) 2022-11-23T02:13:37.7453882Z 2022-11-23T02:13:37.7454147Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7454251Z Ran 1 test in 0.005s 2022-11-23T02:13:37.7454257Z 2022-11-23T02:13:37.7454350Z OK (skipped=1) 2022-11-23T02:13:37.7454356Z 2022-11-23T02:13:37.7454468Z Generating XML reports... 2022-11-23T02:13:37.7454902Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015518.xml 2022-11-23T02:13:37.7455200Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7455565Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7455726Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7456104Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7456281Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7456291Z 2022-11-23T02:13:37.7456392Z Running tests... 2022-11-23T02:13:37.7456655Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7457097Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.003s) 2022-11-23T02:13:37.7457103Z 2022-11-23T02:13:37.7457365Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7457464Z Ran 1 test in 0.003s 2022-11-23T02:13:37.7457470Z 2022-11-23T02:13:37.7457564Z OK (skipped=1) 2022-11-23T02:13:37.7457570Z 2022-11-23T02:13:37.7457687Z Generating XML reports... 2022-11-23T02:13:37.7458119Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015522.xml 2022-11-23T02:13:37.7458430Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7458804Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7459022Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7459403Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7459577Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7459583Z 2022-11-23T02:13:37.7459679Z Running tests... 2022-11-23T02:13:37.7459945Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7460849Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78685 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.586s) 2022-11-23T02:13:37.7460860Z 2022-11-23T02:13:37.7461170Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7461273Z Ran 1 test in 0.586s 2022-11-23T02:13:37.7461279Z 2022-11-23T02:13:37.7461376Z OK (skipped=1) 2022-11-23T02:13:37.7461382Z 2022-11-23T02:13:37.7461495Z Generating XML reports... 2022-11-23T02:13:37.7461929Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015526.xml 2022-11-23T02:13:37.7462228Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7462595Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7462758Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7463136Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7463319Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7463328Z 2022-11-23T02:13:37.7463425Z Running tests... 2022-11-23T02:13:37.7463686Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7464593Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.596s) 2022-11-23T02:13:37.7464600Z 2022-11-23T02:13:37.7464862Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7464963Z Ran 1 test in 0.596s 2022-11-23T02:13:37.7464969Z 2022-11-23T02:13:37.7465062Z OK (skipped=1) 2022-11-23T02:13:37.7465068Z 2022-11-23T02:13:37.7465180Z Generating XML reports... 2022-11-23T02:13:37.7465624Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015531.xml 2022-11-23T02:13:37.7465932Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7466302Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7466465Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7466842Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7467016Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7467022Z 2022-11-23T02:13:37.7467119Z Running tests... 2022-11-23T02:13:37.7467380Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7467713Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24593 2022-11-23T02:13:37.7468002Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24594 2022-11-23T02:13:37.7468253Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7468625Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7468787Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7469166Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7469333Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7469555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7469968Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7470139Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7470519Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7470694Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7470921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7471316Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7471700Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7471913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7472130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7472356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:13:37.7472576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:13:37.7472970Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.7473359Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.7473590Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp34ziqd96 2022-11-23T02:13:37.7473833Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp34ziqd96/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7474067Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbpenmpkb 2022-11-23T02:13:37.7474320Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbpenmpkb/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7474543Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7474761Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7474983Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7475197Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7475279Z ok (8.012s) 2022-11-23T02:13:37.7475293Z 2022-11-23T02:13:37.7475551Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7475651Z Ran 1 test in 8.012s 2022-11-23T02:13:37.7475657Z 2022-11-23T02:13:37.7475739Z OK 2022-11-23T02:13:37.7475745Z 2022-11-23T02:13:37.7475855Z Generating XML reports... 2022-11-23T02:13:37.7476296Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015536.xml 2022-11-23T02:13:37.7476665Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7477033Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7477195Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7477573Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7477748Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7477754Z 2022-11-23T02:13:37.7477851Z Running tests... 2022-11-23T02:13:37.7478115Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7478429Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24816 2022-11-23T02:13:37.7478686Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24817 2022-11-23T02:13:37.7478941Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7479309Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7479471Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7479848Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7480026Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7480248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7480643Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7481017Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7481169Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7481547Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7481723Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7481944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7482340Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7482555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7482814Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-11-23T02:13:37.7483029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7483286Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-11-23T02:13:37.7483517Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr7dopa3n 2022-11-23T02:13:37.7483763Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr7dopa3n/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7483994Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfhnlwuox 2022-11-23T02:13:37.7484239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfhnlwuox/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7484455Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7484665Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7484941Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7485150Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7485405Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-11-23T02:13:37.7485658Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-11-23T02:13:37.7485911Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-11-23T02:13:37.7486169Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-11-23T02:13:37.7486385Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7486597Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7486848Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7487062Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7487316Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-11-23T02:13:37.7487569Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-11-23T02:13:37.7487879Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations 2022-11-23T02:13:37.7488129Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations 2022-11-23T02:13:37.7488343Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7488557Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7488775Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7488987Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7489077Z ok (8.575s) 2022-11-23T02:13:37.7489083Z 2022-11-23T02:13:37.7489357Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7489457Z Ran 1 test in 8.575s 2022-11-23T02:13:37.7489463Z 2022-11-23T02:13:37.7489543Z OK 2022-11-23T02:13:37.7489549Z 2022-11-23T02:13:37.7489662Z Generating XML reports... 2022-11-23T02:13:37.7490099Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015548.xml 2022-11-23T02:13:37.7490407Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7490778Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7490944Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7491326Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7491502Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7491507Z 2022-11-23T02:13:37.7491604Z Running tests... 2022-11-23T02:13:37.7491909Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7492974Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77378 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.636s) 2022-11-23T02:13:37.7492993Z 2022-11-23T02:13:37.7493394Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7493502Z Ran 1 test in 0.636s 2022-11-23T02:13:37.7493509Z 2022-11-23T02:13:37.7493622Z OK (skipped=1) 2022-11-23T02:13:37.7493629Z 2022-11-23T02:13:37.7493761Z Generating XML reports... 2022-11-23T02:13:37.7494278Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015601.xml 2022-11-23T02:13:37.7494644Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7495080Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7495272Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7495719Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7495931Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7496001Z 2022-11-23T02:13:37.7496119Z Running tests... 2022-11-23T02:13:37.7496435Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7496800Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25099 2022-11-23T02:13:37.7497040Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25100 2022-11-23T02:13:37.7497335Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7497767Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7497957Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7498401Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7498618Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7498877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7499344Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7499773Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7499962Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7500400Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7500605Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7500869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7501343Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7501597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7502209Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:13:37.7502462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7503066Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:13:37.7503420Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5s79eq_y 2022-11-23T02:13:37.7503710Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5s79eq_y/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7503985Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgqeygz0l 2022-11-23T02:13:37.7504271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgqeygz0l/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7504527Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7504775Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7505077Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations. 2022-11-23T02:13:37.7505426Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations. 2022-11-23T02:13:37.7505759Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created. 2022-11-23T02:13:37.7506081Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created. 2022-11-23T02:13:37.7506458Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0 2022-11-23T02:13:37.7506773Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively. 2022-11-23T02:13:37.7507074Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0 2022-11-23T02:13:37.7507381Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively. 2022-11-23T02:13:37.7507600Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7507815Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7507906Z ok (7.413s) 2022-11-23T02:13:37.7507912Z 2022-11-23T02:13:37.7508185Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7508283Z Ran 1 test in 7.413s 2022-11-23T02:13:37.7508289Z 2022-11-23T02:13:37.7508370Z OK 2022-11-23T02:13:37.7508376Z 2022-11-23T02:13:37.7508489Z Generating XML reports... 2022-11-23T02:13:37.7508926Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015605.xml 2022-11-23T02:13:37.7509237Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7509599Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7509766Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7510145Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7510324Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7510330Z 2022-11-23T02:13:37.7510426Z Running tests... 2022-11-23T02:13:37.7510695Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7511037Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25316 2022-11-23T02:13:37.7511238Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25317 2022-11-23T02:13:37.7511490Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7511911Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7512075Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7512454Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7512626Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7512846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7513240Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7513605Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7513812Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7514201Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7514376Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7514598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7514993Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7515205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7515421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7515659Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7515891Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1l2uwdsq 2022-11-23T02:13:37.7516138Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1l2uwdsq/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7516390Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7516621Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg0zx48xb 2022-11-23T02:13:37.7516868Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg0zx48xb/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7517083Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7517297Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7517515Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7517728Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7517821Z ok (7.915s) 2022-11-23T02:13:37.7517827Z 2022-11-23T02:13:37.7518095Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7518195Z Ran 1 test in 7.916s 2022-11-23T02:13:37.7518200Z 2022-11-23T02:13:37.7518281Z OK 2022-11-23T02:13:37.7518287Z 2022-11-23T02:13:37.7518397Z Generating XML reports... 2022-11-23T02:13:37.7518833Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015617.xml 2022-11-23T02:13:37.7519146Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7519513Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7519676Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7520057Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7520290Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7520296Z 2022-11-23T02:13:37.7520394Z Running tests... 2022-11-23T02:13:37.7520653Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7521002Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25595 2022-11-23T02:13:37.7521202Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25596 2022-11-23T02:13:37.7521451Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7521815Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7521976Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7522402Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7522582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7522805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7523173Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7523334Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7523713Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7523891Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7524115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7524520Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7524907Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7525119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7525332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7525581Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7525811Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpall5xyxb 2022-11-23T02:13:37.7526054Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpall5xyxb/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7526303Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7526536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfwzvztxa 2022-11-23T02:13:37.7526784Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfwzvztxa/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7526992Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7527201Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7527418Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7527629Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7527828Z ok (7.923s) 2022-11-23T02:13:37.7527834Z 2022-11-23T02:13:37.7528107Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7528208Z Ran 1 test in 7.923s 2022-11-23T02:13:37.7528282Z 2022-11-23T02:13:37.7528371Z OK 2022-11-23T02:13:37.7528377Z 2022-11-23T02:13:37.7528489Z Generating XML reports... 2022-11-23T02:13:37.7528936Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015629.xml 2022-11-23T02:13:37.7529243Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7529610Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7529772Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7530151Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7530325Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7530331Z 2022-11-23T02:13:37.7530426Z Running tests... 2022-11-23T02:13:37.7530743Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7531157Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25874 2022-11-23T02:13:37.7531361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25875 2022-11-23T02:13:37.7531609Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7531982Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7532144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7532511Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7532691Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7532922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7533316Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7533680Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7533843Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7534217Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7534394Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7534614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7535010Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7535229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7535441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7535687Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7535932Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7536162Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpakmcqi1m 2022-11-23T02:13:37.7536392Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8a7rl9jn 2022-11-23T02:13:37.7536636Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpakmcqi1m/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7536883Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8a7rl9jn/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7537156Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7537370Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7537585Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7537799Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7537890Z ok (7.828s) 2022-11-23T02:13:37.7537896Z 2022-11-23T02:13:37.7538158Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7538256Z Ran 1 test in 7.828s 2022-11-23T02:13:37.7538261Z 2022-11-23T02:13:37.7538341Z OK 2022-11-23T02:13:37.7538347Z 2022-11-23T02:13:37.7538459Z Generating XML reports... 2022-11-23T02:13:37.7538948Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015641.xml 2022-11-23T02:13:37.7539271Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7539642Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7539806Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7540183Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7540362Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7540368Z 2022-11-23T02:13:37.7540464Z Running tests... 2022-11-23T02:13:37.7540729Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7541134Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26153 2022-11-23T02:13:37.7541340Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26154 2022-11-23T02:13:37.7541591Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7541955Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7542116Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7542496Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7542671Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7542892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7543289Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7543660Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7543826Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7544193Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7544367Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7544588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7544986Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7545201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7545418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7545717Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7545949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu81ixj7s 2022-11-23T02:13:37.7546191Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu81ixj7s/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7546435Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7546667Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ipdt9rm 2022-11-23T02:13:37.7546914Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ipdt9rm/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7547136Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7547396Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7547621Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7547833Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7547921Z ok (8.729s) 2022-11-23T02:13:37.7547927Z 2022-11-23T02:13:37.7548197Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7548296Z Ran 1 test in 8.730s 2022-11-23T02:13:37.7548302Z 2022-11-23T02:13:37.7548386Z OK 2022-11-23T02:13:37.7548392Z 2022-11-23T02:13:37.7548505Z Generating XML reports... 2022-11-23T02:13:37.7548943Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015653.xml 2022-11-23T02:13:37.7549243Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7549614Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7549783Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7550160Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7550338Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7550344Z 2022-11-23T02:13:37.7550440Z Running tests... 2022-11-23T02:13:37.7550704Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7551105Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26432 2022-11-23T02:13:37.7551308Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26433 2022-11-23T02:13:37.7551563Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7551933Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7552094Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7552476Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7552653Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7552878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7553245Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7553409Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7553790Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7554027Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7554247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7554641Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7555032Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7555246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7555458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7555697Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7555979Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4_mhw1rr 2022-11-23T02:13:37.7556226Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4_mhw1rr/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7556474Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7556708Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpss4wqaml 2022-11-23T02:13:37.7556951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpss4wqaml/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7557173Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7557392Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7557609Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7557824Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7557917Z ok (7.973s) 2022-11-23T02:13:37.7557923Z 2022-11-23T02:13:37.7558191Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7558292Z Ran 1 test in 7.974s 2022-11-23T02:13:37.7558298Z 2022-11-23T02:13:37.7558381Z OK 2022-11-23T02:13:37.7558386Z 2022-11-23T02:13:37.7558501Z Generating XML reports... 2022-11-23T02:13:37.7558939Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015706.xml 2022-11-23T02:13:37.7559248Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7559616Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7559780Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7560159Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7560341Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7560347Z 2022-11-23T02:13:37.7560444Z Running tests... 2022-11-23T02:13:37.7560700Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7561100Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26711 2022-11-23T02:13:37.7561310Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26712 2022-11-23T02:13:37.7561558Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7561924Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7562156Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7562535Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7562713Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7562935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7563331Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7563696Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7563860Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7564240Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7564483Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7564714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7565110Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7565325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7565539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7565789Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7566019Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsaji0e1j 2022-11-23T02:13:37.7566264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsaji0e1j/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7566520Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7566755Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphnnkd4nu 2022-11-23T02:13:37.7566998Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphnnkd4nu/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7567203Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7567414Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7567627Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7567876Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7567967Z ok (7.947s) 2022-11-23T02:13:37.7567973Z 2022-11-23T02:13:37.7568241Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7568347Z Ran 1 test in 7.948s 2022-11-23T02:13:37.7568354Z 2022-11-23T02:13:37.7568437Z OK 2022-11-23T02:13:37.7568443Z 2022-11-23T02:13:37.7568554Z Generating XML reports... 2022-11-23T02:13:37.7568990Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015718.xml 2022-11-23T02:13:37.7569298Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7569663Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7569825Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7570204Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7570382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7570388Z 2022-11-23T02:13:37.7570553Z Running tests... 2022-11-23T02:13:37.7570830Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7571234Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26990 2022-11-23T02:13:37.7571437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26991 2022-11-23T02:13:37.7571686Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7572054Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7572214Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7572591Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7572817Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7573042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7573414Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7573575Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7573951Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7574126Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7574350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7574743Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7575135Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7575352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7575565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7575814Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7576044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy_x5rh6u 2022-11-23T02:13:37.7576286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy_x5rh6u/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7576534Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7576765Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeo8_bxhd 2022-11-23T02:13:37.7577013Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeo8_bxhd/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7577231Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7577444Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7577658Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7577871Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7577962Z ok (7.970s) 2022-11-23T02:13:37.7577968Z 2022-11-23T02:13:37.7578235Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7578325Z Ran 1 test in 7.971s 2022-11-23T02:13:37.7578331Z 2022-11-23T02:13:37.7578414Z OK 2022-11-23T02:13:37.7578419Z 2022-11-23T02:13:37.7578530Z Generating XML reports... 2022-11-23T02:13:37.7579027Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015730.xml 2022-11-23T02:13:37.7579336Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7579701Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7579862Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7580242Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7580421Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7580427Z 2022-11-23T02:13:37.7580525Z Running tests... 2022-11-23T02:13:37.7580791Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7581238Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27269 2022-11-23T02:13:37.7581450Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27270 2022-11-23T02:13:37.7581699Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7582069Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7582231Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7582609Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7582784Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7583008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7583384Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7583546Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7583923Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7584101Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7584313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7584710Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7585095Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7585308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7585532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7585778Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7586025Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7586256Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmped7fs62c 2022-11-23T02:13:37.7586501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmped7fs62c/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7586731Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3pj9wqto 2022-11-23T02:13:37.7586974Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3pj9wqto/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7587192Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7587462Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7587677Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7587890Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7587979Z ok (7.614s) 2022-11-23T02:13:37.7587985Z 2022-11-23T02:13:37.7588254Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7588357Z Ran 1 test in 7.614s 2022-11-23T02:13:37.7588363Z 2022-11-23T02:13:37.7588444Z OK 2022-11-23T02:13:37.7588449Z 2022-11-23T02:13:37.7588561Z Generating XML reports... 2022-11-23T02:13:37.7589000Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015742.xml 2022-11-23T02:13:37.7589312Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7589721Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7589887Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7590271Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7590448Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7590453Z 2022-11-23T02:13:37.7590551Z Running tests... 2022-11-23T02:13:37.7590815Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7591213Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27548 2022-11-23T02:13:37.7591415Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27549 2022-11-23T02:13:37.7591671Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7592036Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7592198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7592574Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7592748Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7592975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7593343Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7593505Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7593891Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7594067Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7594289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7594687Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7595077Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7595292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7595505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7595755Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7596033Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpse4oqebz 2022-11-23T02:13:37.7596278Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpse4oqebz/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7596526Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7596756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_67r_q25 2022-11-23T02:13:37.7596991Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_67r_q25/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7597209Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7597420Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7597676Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7597892Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7597988Z ok (7.915s) 2022-11-23T02:13:37.7597994Z 2022-11-23T02:13:37.7598268Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7598368Z Ran 1 test in 7.916s 2022-11-23T02:13:37.7598374Z 2022-11-23T02:13:37.7598455Z OK 2022-11-23T02:13:37.7598461Z 2022-11-23T02:13:37.7598573Z Generating XML reports... 2022-11-23T02:13:37.7599011Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015754.xml 2022-11-23T02:13:37.7599321Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7599690Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7599854Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7600240Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7600415Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7600421Z 2022-11-23T02:13:37.7600517Z Running tests... 2022-11-23T02:13:37.7600777Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7601176Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27827 2022-11-23T02:13:37.7601377Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27828 2022-11-23T02:13:37.7601623Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7601994Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7602161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7602539Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7602716Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7602943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7603308Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7603475Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7603851Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7604031Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7604316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7604713Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7605099Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7605313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7605527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7605777Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7606007Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4cu690ih 2022-11-23T02:13:37.7606294Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4cu690ih/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7606548Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7606784Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz1frhsoi 2022-11-23T02:13:37.7607027Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz1frhsoi/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7607244Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7607447Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7607662Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7607910Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7608000Z ok (7.922s) 2022-11-23T02:13:37.7608006Z 2022-11-23T02:13:37.7608287Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7608390Z Ran 1 test in 7.923s 2022-11-23T02:13:37.7608395Z 2022-11-23T02:13:37.7608478Z OK 2022-11-23T02:13:37.7608484Z 2022-11-23T02:13:37.7608594Z Generating XML reports... 2022-11-23T02:13:37.7609036Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015806.xml 2022-11-23T02:13:37.7609346Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7609711Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7609872Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7610255Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7610432Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7610445Z 2022-11-23T02:13:37.7610548Z Running tests... 2022-11-23T02:13:37.7610812Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7611159Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28106 2022-11-23T02:13:37.7611365Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28107 2022-11-23T02:13:37.7611611Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7611978Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7612138Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7612519Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7612756Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7612976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7613346Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7613509Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7613890Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7614069Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7614293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7614685Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7615129Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7615342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7615554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7615804Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7616038Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp20wamq4e 2022-11-23T02:13:37.7616279Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp20wamq4e/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7616526Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7616759Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr9mzerak 2022-11-23T02:13:37.7617008Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr9mzerak/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7617225Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7617436Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7617652Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7617863Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7617956Z ok (7.917s) 2022-11-23T02:13:37.7617962Z 2022-11-23T02:13:37.7618234Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7618325Z Ran 1 test in 7.918s 2022-11-23T02:13:37.7618331Z 2022-11-23T02:13:37.7618412Z OK 2022-11-23T02:13:37.7618418Z 2022-11-23T02:13:37.7618531Z Generating XML reports... 2022-11-23T02:13:37.7618973Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015818.xml 2022-11-23T02:13:37.7619282Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7619648Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7619813Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7620190Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7620366Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7620372Z 2022-11-23T02:13:37.7620470Z Running tests... 2022-11-23T02:13:37.7620733Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7621079Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28385 2022-11-23T02:13:37.7621344Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28386 2022-11-23T02:13:37.7621592Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7621967Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7622130Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7622508Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7622684Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7622909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7623329Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7623498Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7623881Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7624049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7624271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7624665Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7625053Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7625268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7625485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7625732Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7625963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkihc_uat 2022-11-23T02:13:37.7626205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkihc_uat/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7626451Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7626679Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_274nr7j 2022-11-23T02:13:37.7626917Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_274nr7j/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7627135Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7627351Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7627566Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7627777Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7627866Z ok (7.837s) 2022-11-23T02:13:37.7627872Z 2022-11-23T02:13:37.7628143Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7628242Z Ran 1 test in 7.838s 2022-11-23T02:13:37.7628248Z 2022-11-23T02:13:37.7628330Z OK 2022-11-23T02:13:37.7628335Z 2022-11-23T02:13:37.7628447Z Generating XML reports... 2022-11-23T02:13:37.7628887Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015830.xml 2022-11-23T02:13:37.7629185Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7629629Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7629797Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7630177Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7630355Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7630361Z 2022-11-23T02:13:37.7630459Z Running tests... 2022-11-23T02:13:37.7630724Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7631150Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'ucc', 'gloo'} (0.001s) 2022-11-23T02:13:37.7631157Z 2022-11-23T02:13:37.7631418Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7631516Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7631526Z 2022-11-23T02:13:37.7631689Z OK (skipped=1) 2022-11-23T02:13:37.7631696Z 2022-11-23T02:13:37.7631815Z Generating XML reports... 2022-11-23T02:13:37.7632256Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015842.xml 2022-11-23T02:13:37.7632568Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7632938Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7633104Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7633482Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7633660Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7633666Z 2022-11-23T02:13:37.7633763Z Running tests... 2022-11-23T02:13:37.7634036Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7634327Z test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28730 2022-11-23T02:13:37.7634530Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28731 2022-11-23T02:13:37.7634781Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7635147Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7635302Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7635681Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7635857Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7636088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7636455Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7636619Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7636996Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7637172Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7637394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7637789Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7638177Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7638449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7638659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7638891Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj9hkmxjt 2022-11-23T02:13:37.7639134Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj9hkmxjt/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7639366Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl08h9c45 2022-11-23T02:13:37.7639609Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl08h9c45/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7639702Z ok (7.923s) 2022-11-23T02:13:37.7639708Z 2022-11-23T02:13:37.7639975Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7640075Z Ran 1 test in 7.924s 2022-11-23T02:13:37.7640081Z 2022-11-23T02:13:37.7640162Z OK 2022-11-23T02:13:37.7640171Z 2022-11-23T02:13:37.7640327Z Generating XML reports... 2022-11-23T02:13:37.7640760Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015846.xml 2022-11-23T02:13:37.7641068Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7641436Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7641599Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7641977Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7642153Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7642159Z 2022-11-23T02:13:37.7642255Z Running tests... 2022-11-23T02:13:37.7642520Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7642836Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28940 2022-11-23T02:13:37.7643038Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28941 2022-11-23T02:13:37.7643290Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7643656Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7643819Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7644196Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7644373Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7644601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7644974Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7645139Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7645515Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7645691Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7645915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7646312Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7646686Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7646902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7647170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7647401Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgs40s4vv 2022-11-23T02:13:37.7647647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgs40s4vv/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7647969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvg0deur4 2022-11-23T02:13:37.7648214Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvg0deur4/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7648432Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7648644Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7649110Z /opt/conda/lib/python3.8/tempfile.py:818: ResourceWarning: Implicitly cleaning up 2022-11-23T02:13:37.7649269Z _warnings.warn(warn_message, ResourceWarning) 2022-11-23T02:13:37.7649666Z /opt/conda/lib/python3.8/tempfile.py:818: ResourceWarning: Implicitly cleaning up 2022-11-23T02:13:37.7649814Z _warnings.warn(warn_message, ResourceWarning) 2022-11-23T02:13:37.7649905Z ok (7.420s) 2022-11-23T02:13:37.7649911Z 2022-11-23T02:13:37.7650176Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7650276Z Ran 1 test in 7.421s 2022-11-23T02:13:37.7650282Z 2022-11-23T02:13:37.7650365Z OK 2022-11-23T02:13:37.7650371Z 2022-11-23T02:13:37.7650484Z Generating XML reports... 2022-11-23T02:13:37.7650925Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015858.xml 2022-11-23T02:13:37.7651237Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7651608Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7651776Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7652145Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7652320Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7652325Z 2022-11-23T02:13:37.7652425Z Running tests... 2022-11-23T02:13:37.7652685Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7652987Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29157 2022-11-23T02:13:37.7653190Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29158 2022-11-23T02:13:37.7653439Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7653809Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7653969Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7654347Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7654525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7654749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7655143Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7655507Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7655671Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7656116Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7656293Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7656517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7656914Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7657128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7657360Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb4f0pmi4 2022-11-23T02:13:37.7657605Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb4f0pmi4/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7657808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7658162Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0wap879i 2022-11-23T02:13:37.7658409Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0wap879i/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7658625Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7658837Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7658927Z ok (5.470s) 2022-11-23T02:13:37.7658933Z 2022-11-23T02:13:37.7659205Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7659304Z Ran 1 test in 5.471s 2022-11-23T02:13:37.7659310Z 2022-11-23T02:13:37.7659395Z OK 2022-11-23T02:13:37.7659401Z 2022-11-23T02:13:37.7659515Z Generating XML reports... 2022-11-23T02:13:37.7659951Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015909.xml 2022-11-23T02:13:37.7660267Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7660634Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7660796Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7661178Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7661354Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7661360Z 2022-11-23T02:13:37.7661456Z Running tests... 2022-11-23T02:13:37.7661723Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7662025Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29430 2022-11-23T02:13:37.7662232Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29431 2022-11-23T02:13:37.7662485Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7662853Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7663005Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7663385Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7663566Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7663789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7664156Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7664320Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7664758Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7664934Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7665155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7665552Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7665940Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7666151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7666365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7666642Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxz2h4k0t 2022-11-23T02:13:37.7666894Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxz2h4k0t/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7667126Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpucztmrjj 2022-11-23T02:13:37.7667372Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpucztmrjj/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7667594Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7667804Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7667894Z ok (7.664s) 2022-11-23T02:13:37.7667900Z 2022-11-23T02:13:37.7668172Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7668272Z Ran 1 test in 7.665s 2022-11-23T02:13:37.7668278Z 2022-11-23T02:13:37.7668351Z OK 2022-11-23T02:13:37.7668366Z 2022-11-23T02:13:37.7668469Z Generating XML reports... 2022-11-23T02:13:37.7668912Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015919.xml 2022-11-23T02:13:37.7669224Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7669597Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7669762Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7670138Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7670312Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7670318Z 2022-11-23T02:13:37.7670415Z Running tests... 2022-11-23T02:13:37.7670677Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7671143Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.002s) 2022-11-23T02:13:37.7671153Z 2022-11-23T02:13:37.7671415Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7671515Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7671521Z 2022-11-23T02:13:37.7671617Z OK (skipped=1) 2022-11-23T02:13:37.7671622Z 2022-11-23T02:13:37.7671736Z Generating XML reports... 2022-11-23T02:13:37.7672171Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015931.xml 2022-11-23T02:13:37.7672483Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7672850Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7673013Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7673393Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7673626Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7673632Z 2022-11-23T02:13:37.7673734Z Running tests... 2022-11-23T02:13:37.7674002Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7674442Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.002s) 2022-11-23T02:13:37.7674461Z 2022-11-23T02:13:37.7674711Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7674812Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7674818Z 2022-11-23T02:13:37.7674915Z OK (skipped=1) 2022-11-23T02:13:37.7674921Z 2022-11-23T02:13:37.7675034Z Generating XML reports... 2022-11-23T02:13:37.7675518Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015935.xml 2022-11-23T02:13:37.7675836Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7676201Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7676365Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7676744Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7676920Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7676925Z 2022-11-23T02:13:37.7677024Z Running tests... 2022-11-23T02:13:37.7677289Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7677780Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'ucc', 'gloo'} (0.001s) 2022-11-23T02:13:37.7677796Z 2022-11-23T02:13:37.7678055Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7678154Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7678159Z 2022-11-23T02:13:37.7678256Z OK (skipped=1) 2022-11-23T02:13:37.7678262Z 2022-11-23T02:13:37.7678375Z Generating XML reports... 2022-11-23T02:13:37.7678811Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015939.xml 2022-11-23T02:13:37.7679118Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7679486Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7679646Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7680023Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7680206Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7680212Z 2022-11-23T02:13:37.7680300Z Running tests... 2022-11-23T02:13:37.7680565Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7681032Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.001s) 2022-11-23T02:13:37.7681038Z 2022-11-23T02:13:37.7681302Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7681401Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7681408Z 2022-11-23T02:13:37.7681509Z OK (skipped=1) 2022-11-23T02:13:37.7681515Z 2022-11-23T02:13:37.7681627Z Generating XML reports... 2022-11-23T02:13:37.7682059Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015943.xml 2022-11-23T02:13:37.7682373Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7682800Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7682964Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7683346Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7683523Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7683529Z 2022-11-23T02:13:37.7683626Z Running tests... 2022-11-23T02:13:37.7683892Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7684310Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.003s) 2022-11-23T02:13:37.7684317Z 2022-11-23T02:13:37.7684579Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7684682Z Ran 1 test in 0.003s 2022-11-23T02:13:37.7684733Z 2022-11-23T02:13:37.7684838Z OK (skipped=1) 2022-11-23T02:13:37.7684844Z 2022-11-23T02:13:37.7684957Z Generating XML reports... 2022-11-23T02:13:37.7685397Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015947.xml 2022-11-23T02:13:37.7685709Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7686077Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7686228Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7686606Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7686782Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7686787Z 2022-11-23T02:13:37.7686885Z Running tests... 2022-11-23T02:13:37.7687155Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7687451Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29977 2022-11-23T02:13:37.7687656Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29978 2022-11-23T02:13:37.7687948Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7688323Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7688484Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7688862Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7689040Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7689272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7689636Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7689801Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7690176Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7690351Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7690575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7690968Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7691363Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7691959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7692239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7692527Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8jz5w0xb 2022-11-23T02:13:37.7692774Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8jz5w0xb/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7693003Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsz4ira73 2022-11-23T02:13:37.7693242Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsz4ira73/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7694061Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:13:37.7694824Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:13:37.7695042Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7695256Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7695347Z ok (8.025s) 2022-11-23T02:13:37.7695354Z 2022-11-23T02:13:37.7695641Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7695740Z Ran 1 test in 8.025s 2022-11-23T02:13:37.7695746Z 2022-11-23T02:13:37.7695828Z OK 2022-11-23T02:13:37.7695834Z 2022-11-23T02:13:37.7695947Z Generating XML reports... 2022-11-23T02:13:37.7696388Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015951.xml 2022-11-23T02:13:37.7696699Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7697066Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7697230Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7697609Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7697787Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7697793Z 2022-11-23T02:13:37.7697890Z Running tests... 2022-11-23T02:13:37.7698155Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7699091Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78338 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.626s) 2022-11-23T02:13:37.7699099Z 2022-11-23T02:13:37.7699363Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7699462Z Ran 1 test in 0.626s 2022-11-23T02:13:37.7699468Z 2022-11-23T02:13:37.7699621Z OK (skipped=1) 2022-11-23T02:13:37.7699631Z 2022-11-23T02:13:37.7699735Z Generating XML reports... 2022-11-23T02:13:37.7700178Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020003.xml 2022-11-23T02:13:37.7700488Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7700857Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7701020Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7701399Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7701574Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7701580Z 2022-11-23T02:13:37.7701677Z Running tests... 2022-11-23T02:13:37.7701943Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7702442Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.002s) 2022-11-23T02:13:37.7702450Z 2022-11-23T02:13:37.7702718Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7702820Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7702826Z 2022-11-23T02:13:37.7702925Z OK (skipped=1) 2022-11-23T02:13:37.7702930Z 2022-11-23T02:13:37.7703044Z Generating XML reports... 2022-11-23T02:13:37.7703482Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020008.xml 2022-11-23T02:13:37.7703791Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7704159Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7704321Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7704709Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7704886Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7704892Z 2022-11-23T02:13:37.7704990Z Running tests... 2022-11-23T02:13:37.7705256Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7705760Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.002s) 2022-11-23T02:13:37.7705767Z 2022-11-23T02:13:37.7706100Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7706193Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7706268Z 2022-11-23T02:13:37.7706359Z OK (skipped=1) 2022-11-23T02:13:37.7706365Z 2022-11-23T02:13:37.7706547Z Generating XML reports... 2022-11-23T02:13:37.7707057Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020012.xml 2022-11-23T02:13:37.7707435Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7707859Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7708073Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7708521Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7708752Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7708759Z 2022-11-23T02:13:37.7708917Z Running tests... 2022-11-23T02:13:37.7709254Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7709624Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30392 2022-11-23T02:13:37.7709964Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30393 2022-11-23T02:13:37.7710272Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7710863Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7711082Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7711524Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7711769Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7712050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7712541Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7712775Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7713295Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7713464Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7713749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7714214Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7714681Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7714949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7715233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7715525Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp07y6tnrg 2022-11-23T02:13:37.7715824Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp07y6tnrg/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7716113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx1evdyk2 2022-11-23T02:13:37.7716412Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx1evdyk2/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7716573Z ok (5.313s) 2022-11-23T02:13:37.7716579Z 2022-11-23T02:13:37.7716915Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7717066Z Ran 1 test in 5.314s 2022-11-23T02:13:37.7717072Z 2022-11-23T02:13:37.7717206Z OK 2022-11-23T02:13:37.7717212Z 2022-11-23T02:13:37.7717375Z Generating XML reports... 2022-11-23T02:13:37.7717880Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020016.xml 2022-11-23T02:13:37.7718253Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7718686Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7718918Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7719360Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7719589Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7719595Z 2022-11-23T02:13:37.7719746Z Running tests... 2022-11-23T02:13:37.7720009Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7721054Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78595 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.583s) 2022-11-23T02:13:37.7721123Z 2022-11-23T02:13:37.7721460Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7721787Z Ran 1 test in 0.583s 2022-11-23T02:13:37.7721795Z 2022-11-23T02:13:37.7721887Z OK (skipped=1) 2022-11-23T02:13:37.7721951Z 2022-11-23T02:13:37.7722058Z Generating XML reports... 2022-11-23T02:13:37.7722561Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020026.xml 2022-11-23T02:13:37.7722935Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7723362Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7723641Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7724171Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7724413Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7724420Z 2022-11-23T02:13:37.7724571Z Running tests... 2022-11-23T02:13:37.7724901Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7725411Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.003s) 2022-11-23T02:13:37.7725418Z 2022-11-23T02:13:37.7725738Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7725890Z Ran 1 test in 0.003s 2022-11-23T02:13:37.7725896Z 2022-11-23T02:13:37.7726045Z OK (skipped=1) 2022-11-23T02:13:37.7726051Z 2022-11-23T02:13:37.7726230Z Generating XML reports... 2022-11-23T02:13:37.7726742Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020030.xml 2022-11-23T02:13:37.7727117Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7727547Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7727822Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7728269Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7728508Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7728515Z 2022-11-23T02:13:37.7728667Z Running tests... 2022-11-23T02:13:37.7728928Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7729985Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.581s) 2022-11-23T02:13:37.7729996Z 2022-11-23T02:13:37.7730321Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7730470Z Ran 1 test in 0.581s 2022-11-23T02:13:37.7730476Z 2022-11-23T02:13:37.7730565Z OK (skipped=1) 2022-11-23T02:13:37.7730637Z 2022-11-23T02:13:37.7730744Z Generating XML reports... 2022-11-23T02:13:37.7731241Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020034.xml 2022-11-23T02:13:37.7731609Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7732052Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7732346Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7732952Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7733182Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7733189Z 2022-11-23T02:13:37.7733340Z Running tests... 2022-11-23T02:13:37.7733667Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7734031Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30797 2022-11-23T02:13:37.7734304Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30798 2022-11-23T02:13:37.7734679Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7735184Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7735408Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7735854Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7736082Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7736358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7736799Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7737017Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7737458Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7737693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7737910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7738370Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7738826Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7739105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7739376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7739676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwrm5fe9f 2022-11-23T02:13:37.7739981Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwrm5fe9f/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7740273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpevy3fszk 2022-11-23T02:13:37.7740583Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpevy3fszk/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7740980Z STAGE:2022-11-23 02:00:42 30798:30798 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7741371Z STAGE:2022-11-23 02:00:42 30797:30797 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7741640Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7741923Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:13:37.7742335Z STAGE:2022-11-23 02:00:42 30798:30798 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7742740Z STAGE:2022-11-23 02:00:42 30797:30797 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7743159Z STAGE:2022-11-23 02:00:42 30798:30798 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7743655Z STAGE:2022-11-23 02:00:42 30797:30797 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7744073Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 5 (function operator()) 2022-11-23T02:13:37.7744354Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 5 (function operator()) 2022-11-23T02:13:37.7744760Z STAGE:2022-11-23 02:00:42 30797:30797 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7745153Z STAGE:2022-11-23 02:00:43 30797:30797 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7745629Z STAGE:2022-11-23 02:00:43 30797:30797 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7745772Z ok (5.808s) 2022-11-23T02:13:37.7745779Z 2022-11-23T02:13:37.7746108Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7746273Z Ran 1 test in 5.808s 2022-11-23T02:13:37.7746281Z 2022-11-23T02:13:37.7746423Z OK 2022-11-23T02:13:37.7746429Z 2022-11-23T02:13:37.7746596Z Generating XML reports... 2022-11-23T02:13:37.7747113Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020039.xml 2022-11-23T02:13:37.7747486Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7747918Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7748135Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7748575Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7748804Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7748810Z 2022-11-23T02:13:37.7748963Z Running tests... 2022-11-23T02:13:37.7749291Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7749662Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31018 2022-11-23T02:13:37.7749923Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31019 2022-11-23T02:13:37.7750229Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7750659Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7750872Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7751314Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7751549Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7751846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7752275Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7752493Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7752935Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7753102Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7753375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7753834Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7754290Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7754792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7755059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7755347Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv1wucnwg 2022-11-23T02:13:37.7755649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv1wucnwg/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7756004Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9hluxd4i 2022-11-23T02:13:37.7756303Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9hluxd4i/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7756446Z ok (5.613s) 2022-11-23T02:13:37.7756453Z 2022-11-23T02:13:37.7756791Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7756951Z Ran 1 test in 5.614s 2022-11-23T02:13:37.7756963Z 2022-11-23T02:13:37.7757150Z OK 2022-11-23T02:13:37.7757157Z 2022-11-23T02:13:37.7757339Z Generating XML reports... 2022-11-23T02:13:37.7757852Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020049.xml 2022-11-23T02:13:37.7758226Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7758656Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7758876Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7759329Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7759556Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7759562Z 2022-11-23T02:13:37.7759710Z Running tests... 2022-11-23T02:13:37.7759977Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7760349Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31229 2022-11-23T02:13:37.7760607Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31230 2022-11-23T02:13:37.7760914Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7761341Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7761567Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7762007Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7762234Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7762514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7762947Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7763163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7763602Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7763828Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7764116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7764569Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7765090Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7765434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7765766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7766054Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5yrstc7n 2022-11-23T02:13:37.7766359Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5yrstc7n/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7766658Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpngpkcunw 2022-11-23T02:13:37.7766964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpngpkcunw/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7767047Z ok (5.314s) 2022-11-23T02:13:37.7767116Z 2022-11-23T02:13:37.7767386Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7767543Z Ran 1 test in 5.315s 2022-11-23T02:13:37.7767549Z 2022-11-23T02:13:37.7767683Z OK 2022-11-23T02:13:37.7767868Z 2022-11-23T02:13:37.7768116Z Generating XML reports... 2022-11-23T02:13:37.7768646Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020059.xml 2022-11-23T02:13:37.7769028Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7769458Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7769675Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7770112Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7770343Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7770350Z 2022-11-23T02:13:37.7770501Z Running tests... 2022-11-23T02:13:37.7770830Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7772934Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78684 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.581s) 2022-11-23T02:13:37.7772949Z 2022-11-23T02:13:37.7773648Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7774453Z Ran 1 test in 0.581s 2022-11-23T02:13:37.7774785Z 2022-11-23T02:13:37.7775030Z OK (skipped=1) 2022-11-23T02:13:37.7775453Z 2022-11-23T02:13:37.7775772Z Generating XML reports... 2022-11-23T02:13:37.7776947Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020108.xml 2022-11-23T02:13:37.7778250Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7779556Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7780564Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7781730Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7782537Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7782936Z 2022-11-23T02:13:37.7783212Z Running tests... 2022-11-23T02:13:37.7784069Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7786152Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.587s) 2022-11-23T02:13:37.7787287Z 2022-11-23T02:13:37.7787913Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7788640Z Ran 1 test in 0.588s 2022-11-23T02:13:37.7788969Z 2022-11-23T02:13:37.7789210Z OK (skipped=1) 2022-11-23T02:13:37.7789531Z 2022-11-23T02:13:37.7789822Z Generating XML reports... 2022-11-23T02:13:37.7791047Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020113.xml 2022-11-23T02:13:37.7792333Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7793609Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7794385Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7795579Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7796563Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7797004Z 2022-11-23T02:13:37.7797340Z Running tests... 2022-11-23T02:13:37.7798323Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7800439Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.609s) 2022-11-23T02:13:37.7801483Z 2022-11-23T02:13:37.7802102Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7802844Z Ran 1 test in 0.609s 2022-11-23T02:13:37.7803182Z 2022-11-23T02:13:37.7803436Z OK (skipped=1) 2022-11-23T02:13:37.7803674Z 2022-11-23T02:13:37.7803956Z Generating XML reports... 2022-11-23T02:13:37.7805314Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020117.xml 2022-11-23T02:13:37.7806982Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7808688Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7809977Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7811470Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7812567Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7813058Z 2022-11-23T02:13:37.7813334Z Running tests... 2022-11-23T02:13:37.7814167Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7815673Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.003s) 2022-11-23T02:13:37.7816394Z 2022-11-23T02:13:37.7817028Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7817887Z Ran 1 test in 0.003s 2022-11-23T02:13:37.7818192Z 2022-11-23T02:13:37.7818415Z OK (skipped=1) 2022-11-23T02:13:37.7818700Z 2022-11-23T02:13:37.7818942Z Generating XML reports... 2022-11-23T02:13:37.7820013Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020122.xml 2022-11-23T02:13:37.7821128Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7822099Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7822902Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7824071Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7824904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7825285Z 2022-11-23T02:13:37.7825503Z Running tests... 2022-11-23T02:13:37.7826298Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7827375Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31700 2022-11-23T02:13:37.7828344Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31701 2022-11-23T02:13:37.7829378Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7830754Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7831684Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7832750Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7833719Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7834514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7835637Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7836437Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7837353Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7838175Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7838977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7840124Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7841292Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7842164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7842992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7844303Z /opt/conda/lib/python3.8/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op 2022-11-23T02:13:37.7845500Z warnings.warn("Initializing zero-element tensors is a no-op") 2022-11-23T02:13:37.7846251Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzzer_p1t 2022-11-23T02:13:37.7847298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzzer_p1t/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7848668Z /opt/conda/lib/python3.8/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op 2022-11-23T02:13:37.7849700Z warnings.warn("Initializing zero-element tensors is a no-op") 2022-11-23T02:13:37.7850581Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcl7rbnfd 2022-11-23T02:13:37.7851518Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcl7rbnfd/_remote_module_non_scriptable.py 2022-11-23T02:13:37.7852209Z ok (5.307s) 2022-11-23T02:13:37.7852386Z 2022-11-23T02:13:37.7852876Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7853497Z Ran 1 test in 5.307s 2022-11-23T02:13:37.7853805Z 2022-11-23T02:13:37.7854002Z OK 2022-11-23T02:13:37.7854257Z 2022-11-23T02:13:37.7854495Z Generating XML reports... 2022-11-23T02:13:37.7855587Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020126.xml 2022-11-23T02:13:37.7856968Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7858285Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7859247Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7860319Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7861464Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7861955Z 2022-11-23T02:13:37.7862206Z Running tests... 2022-11-23T02:13:37.7863153Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7864344Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31907 2022-11-23T02:13:37.7865762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31908 2022-11-23T02:13:37.7866802Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7868000Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7868942Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7870168Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7871234Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7872219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7873553Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7875342Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7876339Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7877486Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7878369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7879339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7880658Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7881649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7882631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7883793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:13:37.7885067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:13:37.7886164Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.7887831Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.7888775Z ok (5.039s) 2022-11-23T02:13:37.7889167Z 2022-11-23T02:13:37.7889850Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7890457Z Ran 1 test in 5.039s 2022-11-23T02:13:37.7890738Z 2022-11-23T02:13:37.7890959Z OK 2022-11-23T02:13:37.7891206Z 2022-11-23T02:13:37.7891452Z Generating XML reports... 2022-11-23T02:13:37.7892269Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020136.xml 2022-11-23T02:13:37.7893583Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7894609Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7895382Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7896308Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7897043Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7897445Z 2022-11-23T02:13:37.7897688Z Running tests... 2022-11-23T02:13:37.7898426Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7899188Z test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32120 2022-11-23T02:13:37.7900190Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32121 2022-11-23T02:13:37.7900991Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7901996Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7902703Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7903587Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7904314Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7905371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7906567Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7907413Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7908291Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7909037Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7909785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7910745Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7911637Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7912228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7912846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7913545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:13:37.7914263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:13:37.7915095Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.7915947Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.7916491Z ok (5.107s) 2022-11-23T02:13:37.7916712Z 2022-11-23T02:13:37.7917069Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7917469Z Ran 1 test in 5.108s 2022-11-23T02:13:37.7917699Z 2022-11-23T02:13:37.7917863Z OK 2022-11-23T02:13:37.7918069Z 2022-11-23T02:13:37.7918263Z Generating XML reports... 2022-11-23T02:13:37.7919038Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020145.xml 2022-11-23T02:13:37.7919969Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7920762Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7921279Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7922073Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7922709Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7923033Z 2022-11-23T02:13:37.7923217Z Running tests... 2022-11-23T02:13:37.7923886Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7925310Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78767 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.581s) 2022-11-23T02:13:37.7926088Z 2022-11-23T02:13:37.7926478Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7927001Z Ran 1 test in 0.581s 2022-11-23T02:13:37.7927253Z 2022-11-23T02:13:37.7927432Z OK (skipped=1) 2022-11-23T02:13:37.7927658Z 2022-11-23T02:13:37.7927958Z Generating XML reports... 2022-11-23T02:13:37.7928789Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020154.xml 2022-11-23T02:13:37.7929617Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7930447Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7931082Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7931857Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7932472Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7932767Z 2022-11-23T02:13:37.7932854Z Running tests... 2022-11-23T02:13:37.7933498Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7934487Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.002s) 2022-11-23T02:13:37.7935028Z 2022-11-23T02:13:37.7935543Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7936267Z Ran 1 test in 0.002s 2022-11-23T02:13:37.7936707Z 2022-11-23T02:13:37.7937138Z OK (skipped=1) 2022-11-23T02:13:37.7937467Z 2022-11-23T02:13:37.7937762Z Generating XML reports... 2022-11-23T02:13:37.7938716Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020159.xml 2022-11-23T02:13:37.7939800Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7940835Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7941660Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7942648Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7943595Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7944000Z 2022-11-23T02:13:37.7944280Z Running tests... 2022-11-23T02:13:37.7945105Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7945820Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32465 2022-11-23T02:13:37.7946805Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32466 2022-11-23T02:13:37.7947605Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7948452Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7949061Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7949844Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7950534Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7951045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7951908Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7952617Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7953393Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7954008Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7954603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7955440Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7956368Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7956989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7957611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7958146Z ok (4.905s) 2022-11-23T02:13:37.7958372Z 2022-11-23T02:13:37.7958747Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7959225Z Ran 1 test in 4.905s 2022-11-23T02:13:37.7959458Z 2022-11-23T02:13:37.7959621Z OK 2022-11-23T02:13:37.7959829Z 2022-11-23T02:13:37.7960031Z Generating XML reports... 2022-11-23T02:13:37.7960723Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020203.xml 2022-11-23T02:13:37.7961600Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7962386Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7962991Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7963915Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7964535Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7964849Z 2022-11-23T02:13:37.7965034Z Running tests... 2022-11-23T02:13:37.7965541Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7966249Z test_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32672 2022-11-23T02:13:37.7966940Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32673 2022-11-23T02:13:37.7967631Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.7968757Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7969386Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7970145Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7970889Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7971398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.7972221Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7972824Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7973644Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7974267Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7974983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.7975878Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7976654Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.7977324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.7978089Z STAGE:2022-11-23 02:02:15 32673:32673 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7978714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.7979464Z STAGE:2022-11-23 02:02:15 32672:32672 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7980227Z STAGE:2022-11-23 02:02:15 32673:32673 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7981091Z STAGE:2022-11-23 02:02:15 32673:32673 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7981944Z STAGE:2022-11-23 02:02:15 32672:32672 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7982637Z STAGE:2022-11-23 02:02:15 32672:32672 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7983428Z STAGE:2022-11-23 02:02:15 32673:32673 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7984233Z STAGE:2022-11-23 02:02:15 32672:32672 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.7985045Z STAGE:2022-11-23 02:02:15 32672:32672 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7985923Z STAGE:2022-11-23 02:02:15 32672:32672 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7986719Z STAGE:2022-11-23 02:02:15 32673:32673 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.7987736Z STAGE:2022-11-23 02:02:15 32673:32673 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.7988254Z ok (5.129s) 2022-11-23T02:13:37.7988685Z 2022-11-23T02:13:37.7989172Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7989860Z Ran 1 test in 5.130s 2022-11-23T02:13:37.7990209Z 2022-11-23T02:13:37.7990733Z OK 2022-11-23T02:13:37.7991042Z 2022-11-23T02:13:37.7991334Z Generating XML reports... 2022-11-23T02:13:37.7992365Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020212.xml 2022-11-23T02:13:37.7993438Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.7994346Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.7995165Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.7996154Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.7997091Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.7997504Z 2022-11-23T02:13:37.7997767Z Running tests... 2022-11-23T02:13:37.7998605Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.7999496Z test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32885 2022-11-23T02:13:37.8000485Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32886 2022-11-23T02:13:37.8001203Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8002277Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8003197Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8004268Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8005124Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8005932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8007018Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8007995Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8008731Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8009597Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8010322Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8011045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8011973Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8012741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8013474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8013934Z ok (5.007s) 2022-11-23T02:13:37.8014280Z 2022-11-23T02:13:37.8014700Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8015289Z Ran 1 test in 5.008s 2022-11-23T02:13:37.8015572Z 2022-11-23T02:13:37.8015792Z OK 2022-11-23T02:13:37.8016054Z 2022-11-23T02:13:37.8016422Z Generating XML reports... 2022-11-23T02:13:37.8017312Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020221.xml 2022-11-23T02:13:37.8018093Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8018996Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8019692Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8020552Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8021262Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8021615Z 2022-11-23T02:13:37.8021843Z Running tests... 2022-11-23T02:13:37.8022570Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8023162Z test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s) 2022-11-23T02:13:37.8023596Z 2022-11-23T02:13:37.8024008Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8024576Z Ran 1 test in 0.002s 2022-11-23T02:13:37.8024958Z 2022-11-23T02:13:37.8025199Z OK (skipped=1) 2022-11-23T02:13:37.8025471Z 2022-11-23T02:13:37.8025778Z Generating XML reports... 2022-11-23T02:13:37.8026694Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020230.xml 2022-11-23T02:13:37.8027656Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8028599Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8029257Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8030124Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8030864Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8031236Z 2022-11-23T02:13:37.8031477Z Running tests... 2022-11-23T02:13:37.8032239Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8033019Z test_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33158 2022-11-23T02:13:37.8033808Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33159 2022-11-23T02:13:37.8034431Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8035354Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8036044Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8036913Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8037630Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8038324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8039272Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8039955Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8040671Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8041382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8042181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8043114Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8044071Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8044830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8045542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8046121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:13:37.8046851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:13:37.8047983Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.8049106Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.8049999Z STAGE:2022-11-23 02:02:37 33158:33158 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8050847Z STAGE:2022-11-23 02:02:37 33159:33159 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8051807Z STAGE:2022-11-23 02:02:37 33159:33159 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8052745Z STAGE:2022-11-23 02:02:37 33159:33159 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8053492Z STAGE:2022-11-23 02:02:37 33158:33158 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8054310Z STAGE:2022-11-23 02:02:37 33158:33158 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8055065Z STAGE:2022-11-23 02:02:37 33159:33159 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8055805Z STAGE:2022-11-23 02:02:37 33158:33158 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8056653Z STAGE:2022-11-23 02:02:37 33158:33158 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8057508Z STAGE:2022-11-23 02:02:37 33158:33158 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8058301Z STAGE:2022-11-23 02:02:37 33159:33159 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8059069Z STAGE:2022-11-23 02:02:37 33159:33159 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8059483Z ok (5.064s) 2022-11-23T02:13:37.8059707Z 2022-11-23T02:13:37.8060059Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8060529Z Ran 1 test in 5.065s 2022-11-23T02:13:37.8060759Z 2022-11-23T02:13:37.8060942Z OK 2022-11-23T02:13:37.8061148Z 2022-11-23T02:13:37.8061336Z Generating XML reports... 2022-11-23T02:13:37.8062122Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020234.xml 2022-11-23T02:13:37.8062919Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8063625Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8064386Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8065137Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8065758Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8066057Z 2022-11-23T02:13:37.8066230Z Running tests... 2022-11-23T02:13:37.8066908Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8067569Z test_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33377 2022-11-23T02:13:37.8068148Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33378 2022-11-23T02:13:37.8068795Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8069625Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8070239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8071008Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8071624Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8072249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8073018Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8073852Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8074431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8075336Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8075940Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8076510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8094581Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8095163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8095638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8096081Z skip: Skipped due to small world size. (4.809s) 2022-11-23T02:13:37.8096306Z 2022-11-23T02:13:37.8096751Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8097161Z Ran 1 test in 4.809s 2022-11-23T02:13:37.8097585Z 2022-11-23T02:13:37.8097717Z OK (skipped=1) 2022-11-23T02:13:37.8097896Z 2022-11-23T02:13:37.8098033Z Generating XML reports... 2022-11-23T02:13:37.8098804Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020243.xml 2022-11-23T02:13:37.8099624Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8100432Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8100968Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8101741Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8102298Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8102554Z 2022-11-23T02:13:37.8102671Z Running tests... 2022-11-23T02:13:37.8103282Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8103900Z test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33584 2022-11-23T02:13:37.8104534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33585 2022-11-23T02:13:37.8105120Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8105923Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8106466Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8107168Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8107722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8108177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8108837Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8109494Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8109937Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8110510Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8110963Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8111411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8112085Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8112686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8113163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8113503Z ok (5.008s) 2022-11-23T02:13:37.8113648Z 2022-11-23T02:13:37.8113926Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8114239Z Ran 1 test in 5.009s 2022-11-23T02:13:37.8114390Z 2022-11-23T02:13:37.8114475Z OK 2022-11-23T02:13:37.8114601Z 2022-11-23T02:13:37.8114716Z Generating XML reports... 2022-11-23T02:13:37.8115346Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020252.xml 2022-11-23T02:13:37.8116015Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8116650Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8117176Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8117777Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8118316Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8118573Z 2022-11-23T02:13:37.8118688Z Running tests... 2022-11-23T02:13:37.8119192Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8120575Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.579s) 2022-11-23T02:13:37.8121284Z 2022-11-23T02:13:37.8121641Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8122022Z Ran 1 test in 0.580s 2022-11-23T02:13:37.8122201Z 2022-11-23T02:13:37.8122315Z OK (skipped=1) 2022-11-23T02:13:37.8122495Z 2022-11-23T02:13:37.8122631Z Generating XML reports... 2022-11-23T02:13:37.8123343Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020301.xml 2022-11-23T02:13:37.8124102Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8124839Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8125353Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8126037Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8126589Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8126868Z 2022-11-23T02:13:37.8126993Z Running tests... 2022-11-23T02:13:37.8127474Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8128357Z test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33857 2022-11-23T02:13:37.8128890Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33858 2022-11-23T02:13:37.8129370Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8130034Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8130480Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8131062Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8131611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8132054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8132683Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8133119Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8133685Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8134150Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8134582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8135237Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8135987Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8136496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8136952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8137399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:13:37.8137888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:13:37.8138536Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.8139208Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.8139596Z ok (5.316s) 2022-11-23T02:13:37.8139733Z 2022-11-23T02:13:37.8140002Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8140324Z Ran 1 test in 5.317s 2022-11-23T02:13:37.8140478Z 2022-11-23T02:13:37.8140570Z OK 2022-11-23T02:13:37.8140682Z 2022-11-23T02:13:37.8140800Z Generating XML reports... 2022-11-23T02:13:37.8141403Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020306.xml 2022-11-23T02:13:37.8142056Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8142677Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8143114Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8143699Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8144155Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8144374Z 2022-11-23T02:13:37.8144471Z Running tests... 2022-11-23T02:13:37.8144889Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8145416Z test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34070 2022-11-23T02:13:37.8145949Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34071 2022-11-23T02:13:37.8146455Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8147134Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8147581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8148146Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8148604Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8149139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8149761Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8150191Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8150791Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8151243Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8151673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8152302Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8153048Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8153554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8154006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8154352Z ok (4.863s) 2022-11-23T02:13:37.8154488Z 2022-11-23T02:13:37.8154757Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8155068Z Ran 1 test in 4.864s 2022-11-23T02:13:37.8155218Z 2022-11-23T02:13:37.8155291Z OK 2022-11-23T02:13:37.8155411Z 2022-11-23T02:13:37.8155523Z Generating XML reports... 2022-11-23T02:13:37.8156130Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020316.xml 2022-11-23T02:13:37.8156763Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8157375Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8157818Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8158384Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8158823Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8159034Z 2022-11-23T02:13:37.8159141Z Running tests... 2022-11-23T02:13:37.8159539Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8160031Z test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34277 2022-11-23T02:13:37.8160539Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34278 2022-11-23T02:13:37.8161030Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8161684Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8162109Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8162686Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8163146Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8163571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8164187Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8164613Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8165179Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8165702Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8166117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8166754Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8167434Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8168061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8168517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8168846Z ok (5.164s) 2022-11-23T02:13:37.8168980Z 2022-11-23T02:13:37.8169247Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8169555Z Ran 1 test in 5.165s 2022-11-23T02:13:37.8169694Z 2022-11-23T02:13:37.8169782Z OK 2022-11-23T02:13:37.8169856Z 2022-11-23T02:13:37.8169971Z Generating XML reports... 2022-11-23T02:13:37.8170424Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020325.xml 2022-11-23T02:13:37.8170734Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8171101Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8171261Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8171648Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8171821Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8171828Z 2022-11-23T02:13:37.8171923Z Running tests... 2022-11-23T02:13:37.8172188Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8172503Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34484 2022-11-23T02:13:37.8172706Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34485 2022-11-23T02:13:37.8172964Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8173332Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8173492Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8173873Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8174047Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8174283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8174653Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8174813Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8175188Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8175353Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8175574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8175965Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8176352Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8176569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8176848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8177077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:13:37.8177295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:13:37.8177695Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.8178078Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.8178168Z ok (5.508s) 2022-11-23T02:13:37.8178174Z 2022-11-23T02:13:37.8178447Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8178543Z Ran 1 test in 5.508s 2022-11-23T02:13:37.8178549Z 2022-11-23T02:13:37.8178633Z OK 2022-11-23T02:13:37.8178698Z 2022-11-23T02:13:37.8178813Z Generating XML reports... 2022-11-23T02:13:37.8179257Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020334.xml 2022-11-23T02:13:37.8179569Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8179945Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8180107Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8180484Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8180658Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8180664Z 2022-11-23T02:13:37.8180759Z Running tests... 2022-11-23T02:13:37.8181047Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8181344Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34697 2022-11-23T02:13:37.8181551Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34698 2022-11-23T02:13:37.8181814Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8182182Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8182339Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8182726Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8182902Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8183126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8183524Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8183891Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8184055Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8184445Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8184617Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8184834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8185225Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8185443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8185712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8185944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:13:37.8186161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:13:37.8186555Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.8186942Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.8187031Z ok (5.209s) 2022-11-23T02:13:37.8187037Z 2022-11-23T02:13:37.8187298Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8187394Z Ran 1 test in 5.209s 2022-11-23T02:13:37.8187400Z 2022-11-23T02:13:37.8187478Z OK 2022-11-23T02:13:37.8187535Z 2022-11-23T02:13:37.8187643Z Generating XML reports... 2022-11-23T02:13:37.8188083Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020344.xml 2022-11-23T02:13:37.8188390Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8188768Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8188942Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8189319Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8189493Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8189499Z 2022-11-23T02:13:37.8189594Z Running tests... 2022-11-23T02:13:37.8189859Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8190293Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.003s) 2022-11-23T02:13:37.8190299Z 2022-11-23T02:13:37.8190567Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8190663Z Ran 1 test in 0.003s 2022-11-23T02:13:37.8190669Z 2022-11-23T02:13:37.8190760Z OK (skipped=1) 2022-11-23T02:13:37.8190768Z 2022-11-23T02:13:37.8190878Z Generating XML reports... 2022-11-23T02:13:37.8191313Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020353.xml 2022-11-23T02:13:37.8191621Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8191999Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8192160Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8192543Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8192718Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8192724Z 2022-11-23T02:13:37.8192823Z Running tests... 2022-11-23T02:13:37.8193080Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8193365Z test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34976 2022-11-23T02:13:37.8193569Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34977 2022-11-23T02:13:37.8193821Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8194192Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8194416Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8194807Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8194984Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8195206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8195571Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8195732Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8196141Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8196326Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8196613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8197014Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8197405Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8197620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8197858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8197951Z ok (5.017s) 2022-11-23T02:13:37.8197957Z 2022-11-23T02:13:37.8198220Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8198318Z Ran 1 test in 5.018s 2022-11-23T02:13:37.8198325Z 2022-11-23T02:13:37.8198406Z OK 2022-11-23T02:13:37.8198411Z 2022-11-23T02:13:37.8198523Z Generating XML reports... 2022-11-23T02:13:37.8198958Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020357.xml 2022-11-23T02:13:37.8199274Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8199654Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8199817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8200198Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8200371Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8200377Z 2022-11-23T02:13:37.8200486Z Running tests... 2022-11-23T02:13:37.8200751Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8201035Z test_isend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35183 2022-11-23T02:13:37.8201243Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35184 2022-11-23T02:13:37.8201499Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8201872Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8202046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8202425Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8202601Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8202822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8203188Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8203416Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8203796Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8203973Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8204192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8204598Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8204986Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8205187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8205396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8205540Z ok (5.808s) 2022-11-23T02:13:37.8205548Z 2022-11-23T02:13:37.8205818Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8205916Z Ran 1 test in 5.808s 2022-11-23T02:13:37.8205922Z 2022-11-23T02:13:37.8206006Z OK 2022-11-23T02:13:37.8206011Z 2022-11-23T02:13:37.8206133Z Generating XML reports... 2022-11-23T02:13:37.8206589Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020406.xml 2022-11-23T02:13:37.8206903Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8207272Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8207432Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8207860Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8208044Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8208050Z 2022-11-23T02:13:37.8208148Z Running tests... 2022-11-23T02:13:37.8208437Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8208750Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35390 2022-11-23T02:13:37.8208956Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35391 2022-11-23T02:13:37.8209222Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8209592Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8209753Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8210137Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8210316Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8210529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8210921Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8211299Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8211459Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8211838Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8212013Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8212246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8212707Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8212917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8213249Z STAGE:2022-11-23 02:04:18 35391:35391 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8213456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8213799Z STAGE:2022-11-23 02:04:19 35390:35390 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8214135Z STAGE:2022-11-23 02:04:19 35391:35391 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8214484Z STAGE:2022-11-23 02:04:19 35391:35391 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8214875Z STAGE:2022-11-23 02:04:19 35390:35390 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8215243Z STAGE:2022-11-23 02:04:19 35390:35390 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8215332Z ok (5.615s) 2022-11-23T02:13:37.8215338Z 2022-11-23T02:13:37.8215599Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8215695Z Ran 1 test in 5.616s 2022-11-23T02:13:37.8215701Z 2022-11-23T02:13:37.8215782Z OK 2022-11-23T02:13:37.8215788Z 2022-11-23T02:13:37.8215898Z Generating XML reports... 2022-11-23T02:13:37.8216336Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020416.xml 2022-11-23T02:13:37.8216653Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8217015Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8217182Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8217558Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8217732Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8217738Z 2022-11-23T02:13:37.8217835Z Running tests... 2022-11-23T02:13:37.8218107Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8218408Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35603 2022-11-23T02:13:37.8218609Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35604 2022-11-23T02:13:37.8218860Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8219229Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8219394Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8219774Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8219947Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8220165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8220540Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8220702Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8221077Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8221250Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8221550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8221944Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8222332Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8222540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8222900Z STAGE:2022-11-23 02:04:28 35604:35604 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8223102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8223428Z STAGE:2022-11-23 02:04:28 35603:35603 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8223813Z STAGE:2022-11-23 02:04:28 35604:35604 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8224193Z STAGE:2022-11-23 02:04:28 35604:35604 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8224526Z STAGE:2022-11-23 02:04:28 35603:35603 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8224901Z STAGE:2022-11-23 02:04:28 35603:35603 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8224991Z ok (5.066s) 2022-11-23T02:13:37.8224997Z 2022-11-23T02:13:37.8225258Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8225354Z Ran 1 test in 5.067s 2022-11-23T02:13:37.8225361Z 2022-11-23T02:13:37.8225459Z OK 2022-11-23T02:13:37.8225465Z 2022-11-23T02:13:37.8225578Z Generating XML reports... 2022-11-23T02:13:37.8226017Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020426.xml 2022-11-23T02:13:37.8226346Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8226716Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8226874Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8227271Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8227446Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8227452Z 2022-11-23T02:13:37.8227546Z Running tests... 2022-11-23T02:13:37.8227808Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8228277Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.002s) 2022-11-23T02:13:37.8228283Z 2022-11-23T02:13:37.8228543Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8228658Z Ran 1 test in 0.002s 2022-11-23T02:13:37.8228663Z 2022-11-23T02:13:37.8228758Z OK (skipped=1) 2022-11-23T02:13:37.8228764Z 2022-11-23T02:13:37.8228873Z Generating XML reports... 2022-11-23T02:13:37.8229298Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020435.xml 2022-11-23T02:13:37.8229612Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8229995Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8230153Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8230531Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8230705Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8230711Z 2022-11-23T02:13:37.8230885Z Running tests... 2022-11-23T02:13:37.8231150Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8231653Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.001s) 2022-11-23T02:13:37.8231660Z 2022-11-23T02:13:37.8231918Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8232014Z Ran 1 test in 0.002s 2022-11-23T02:13:37.8232020Z 2022-11-23T02:13:37.8232114Z OK (skipped=1) 2022-11-23T02:13:37.8232120Z 2022-11-23T02:13:37.8232231Z Generating XML reports... 2022-11-23T02:13:37.8232713Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020439.xml 2022-11-23T02:13:37.8233034Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8233462Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8233631Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8234025Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8234198Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8234204Z 2022-11-23T02:13:37.8234297Z Running tests... 2022-11-23T02:13:37.8234558Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8234894Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35948 2022-11-23T02:13:37.8235097Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35949 2022-11-23T02:13:37.8235336Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8235709Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8235887Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8236261Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8236453Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8236675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8237065Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8237442Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8237601Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8237986Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8238158Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8238396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8238787Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8239014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8239222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8239365Z skip: Skipped due to small world size. (5.014s) 2022-11-23T02:13:37.8239371Z 2022-11-23T02:13:37.8239644Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8239741Z Ran 1 test in 5.015s 2022-11-23T02:13:37.8239824Z 2022-11-23T02:13:37.8239919Z OK (skipped=1) 2022-11-23T02:13:37.8239926Z 2022-11-23T02:13:37.8240036Z Generating XML reports... 2022-11-23T02:13:37.8240476Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020443.xml 2022-11-23T02:13:37.8240786Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8241154Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8241315Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8241684Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8241866Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8241881Z 2022-11-23T02:13:37.8241968Z Running tests... 2022-11-23T02:13:37.8242279Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8242616Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36155 2022-11-23T02:13:37.8242818Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36156 2022-11-23T02:13:37.8243070Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8243450Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8243609Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8243986Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8244168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8244395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8244794Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8245155Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8245317Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8245703Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8245874Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8246094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8246485Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8246704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8246916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8247136Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms 2022-11-23T02:13:37.8247229Z ok (6.912s) 2022-11-23T02:13:37.8247235Z 2022-11-23T02:13:37.8247489Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8247587Z Ran 1 test in 6.913s 2022-11-23T02:13:37.8247593Z 2022-11-23T02:13:37.8247673Z OK 2022-11-23T02:13:37.8247679Z 2022-11-23T02:13:37.8247849Z Generating XML reports... 2022-11-23T02:13:37.8248292Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020452.xml 2022-11-23T02:13:37.8248603Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8249063Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8249225Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8249604Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8249784Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8249790Z 2022-11-23T02:13:37.8249886Z Running tests... 2022-11-23T02:13:37.8250155Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8250475Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36362 2022-11-23T02:13:37.8250689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36363 2022-11-23T02:13:37.8250997Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8251381Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8251546Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8251925Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8252101Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8252322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8252717Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8253080Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8253260Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8253636Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8253820Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8254042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8254431Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8254644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8254861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8255083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:13:37.8255308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:13:37.8255731Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.8256122Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.8256321Z [E ProcessGroupGloo.cpp:137] Rank 0 timed out in monitoredBarrier after 0 ms. 2022-11-23T02:13:37.8256504Z No ranks successfully processed in monitoredBarrier. 2022-11-23T02:13:37.8256722Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms 2022-11-23T02:13:37.8256814Z ok (5.005s) 2022-11-23T02:13:37.8256820Z 2022-11-23T02:13:37.8257088Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8257190Z Ran 1 test in 5.005s 2022-11-23T02:13:37.8257196Z 2022-11-23T02:13:37.8257281Z OK 2022-11-23T02:13:37.8257287Z 2022-11-23T02:13:37.8257468Z Generating XML reports... 2022-11-23T02:13:37.8257920Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020503.xml 2022-11-23T02:13:37.8258236Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8258609Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8258782Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8259151Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8259329Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8259350Z 2022-11-23T02:13:37.8259437Z Running tests... 2022-11-23T02:13:37.8259710Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8260120Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36575 2022-11-23T02:13:37.8260346Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36576 2022-11-23T02:13:37.8260598Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8260978Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8261142Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8261525Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8261702Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8261936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8262316Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8262484Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8262866Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8263053Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8263275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8263671Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8264070Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8264286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8264508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8264729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:13:37.8264959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:13:37.8265355Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.8265743Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.8265944Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms 2022-11-23T02:13:37.8266043Z ok (5.425s) 2022-11-23T02:13:37.8266049Z 2022-11-23T02:13:37.8266319Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8266419Z Ran 1 test in 5.425s 2022-11-23T02:13:37.8266505Z 2022-11-23T02:13:37.8266590Z OK 2022-11-23T02:13:37.8266596Z 2022-11-23T02:13:37.8266713Z Generating XML reports... 2022-11-23T02:13:37.8267162Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020512.xml 2022-11-23T02:13:37.8267481Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8267845Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8268007Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8268394Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8268569Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8268575Z 2022-11-23T02:13:37.8268674Z Running tests... 2022-11-23T02:13:37.8268984Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8269315Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36788 2022-11-23T02:13:37.8269519Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36789 2022-11-23T02:13:37.8269770Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8270145Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8270307Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8270692Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8270867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8271094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8271472Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8271851Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8272011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8272393Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8272572Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8272794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8273191Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8273408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8273617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8273760Z skip: Skipped due to small world size. (4.907s) 2022-11-23T02:13:37.8273766Z 2022-11-23T02:13:37.8274037Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8274132Z Ran 1 test in 4.907s 2022-11-23T02:13:37.8274138Z 2022-11-23T02:13:37.8274230Z OK (skipped=1) 2022-11-23T02:13:37.8274236Z 2022-11-23T02:13:37.8274348Z Generating XML reports... 2022-11-23T02:13:37.8274784Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020522.xml 2022-11-23T02:13:37.8275100Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8275471Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8275689Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8276069Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8276243Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8276249Z 2022-11-23T02:13:37.8276360Z Running tests... 2022-11-23T02:13:37.8276622Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8277025Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s) 2022-11-23T02:13:37.8277032Z 2022-11-23T02:13:37.8277300Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8277387Z Ran 1 test in 0.002s 2022-11-23T02:13:37.8277400Z 2022-11-23T02:13:37.8277486Z OK (skipped=1) 2022-11-23T02:13:37.8277496Z 2022-11-23T02:13:37.8277658Z Generating XML reports... 2022-11-23T02:13:37.8278097Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020531.xml 2022-11-23T02:13:37.8278413Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8278780Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8278944Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8279328Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8279499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8279505Z 2022-11-23T02:13:37.8279600Z Running tests... 2022-11-23T02:13:37.8279869Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8280287Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s) 2022-11-23T02:13:37.8280292Z 2022-11-23T02:13:37.8280560Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8280655Z Ran 1 test in 0.003s 2022-11-23T02:13:37.8280661Z 2022-11-23T02:13:37.8280753Z OK (skipped=1) 2022-11-23T02:13:37.8280758Z 2022-11-23T02:13:37.8280869Z Generating XML reports... 2022-11-23T02:13:37.8281309Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020535.xml 2022-11-23T02:13:37.8281626Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8281995Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8282153Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8282548Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8282724Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8282730Z 2022-11-23T02:13:37.8282830Z Running tests... 2022-11-23T02:13:37.8283084Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8283485Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s) 2022-11-23T02:13:37.8283507Z 2022-11-23T02:13:37.8283758Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8283853Z Ran 1 test in 0.002s 2022-11-23T02:13:37.8283859Z 2022-11-23T02:13:37.8283952Z OK (skipped=1) 2022-11-23T02:13:37.8283958Z 2022-11-23T02:13:37.8284076Z Generating XML reports... 2022-11-23T02:13:37.8284517Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020539.xml 2022-11-23T02:13:37.8284891Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8285261Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8285422Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8285808Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8285983Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8285990Z 2022-11-23T02:13:37.8286085Z Running tests... 2022-11-23T02:13:37.8286347Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8286749Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s) 2022-11-23T02:13:37.8286755Z 2022-11-23T02:13:37.8287064Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8287163Z Ran 1 test in 0.003s 2022-11-23T02:13:37.8287169Z 2022-11-23T02:13:37.8287269Z OK (skipped=1) 2022-11-23T02:13:37.8287275Z 2022-11-23T02:13:37.8287385Z Generating XML reports... 2022-11-23T02:13:37.8287940Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020543.xml 2022-11-23T02:13:37.8288262Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8288632Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8288794Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8289164Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8289343Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8289367Z 2022-11-23T02:13:37.8289455Z Running tests... 2022-11-23T02:13:37.8289717Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8290003Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.002s) 2022-11-23T02:13:37.8290009Z 2022-11-23T02:13:37.8290270Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8290368Z Ran 1 test in 0.002s 2022-11-23T02:13:37.8290374Z 2022-11-23T02:13:37.8290473Z OK (skipped=1) 2022-11-23T02:13:37.8290479Z 2022-11-23T02:13:37.8290589Z Generating XML reports... 2022-11-23T02:13:37.8291023Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020547.xml 2022-11-23T02:13:37.8291332Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8291705Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8291878Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8292257Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8292429Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8292435Z 2022-11-23T02:13:37.8292530Z Running tests... 2022-11-23T02:13:37.8292793Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8293027Z test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:13:37.8293033Z 2022-11-23T02:13:37.8293291Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8293388Z Ran 1 test in 0.002s 2022-11-23T02:13:37.8293394Z 2022-11-23T02:13:37.8293486Z OK (skipped=1) 2022-11-23T02:13:37.8293565Z 2022-11-23T02:13:37.8293694Z Generating XML reports... 2022-11-23T02:13:37.8294138Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020551.xml 2022-11-23T02:13:37.8294444Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8294803Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8294970Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8295348Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8295529Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8295534Z 2022-11-23T02:13:37.8295629Z Running tests... 2022-11-23T02:13:37.8295890Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8296209Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:13:37.8296216Z 2022-11-23T02:13:37.8296482Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8296580Z Ran 1 test in 0.002s 2022-11-23T02:13:37.8296586Z 2022-11-23T02:13:37.8296679Z OK (skipped=1) 2022-11-23T02:13:37.8296684Z 2022-11-23T02:13:37.8296801Z Generating XML reports... 2022-11-23T02:13:37.8297236Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020555.xml 2022-11-23T02:13:37.8297548Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8297914Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8298075Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8298454Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8298631Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8298637Z 2022-11-23T02:13:37.8298731Z Running tests... 2022-11-23T02:13:37.8298999Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8299288Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:13:37.8299294Z 2022-11-23T02:13:37.8299555Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8299659Z Ran 1 test in 0.002s 2022-11-23T02:13:37.8299664Z 2022-11-23T02:13:37.8299757Z OK (skipped=1) 2022-11-23T02:13:37.8299763Z 2022-11-23T02:13:37.8299874Z Generating XML reports... 2022-11-23T02:13:37.8300303Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020559.xml 2022-11-23T02:13:37.8300618Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8300994Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8301157Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8301544Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8301718Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8301723Z 2022-11-23T02:13:37.8301819Z Running tests... 2022-11-23T02:13:37.8302092Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8302431Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37523 2022-11-23T02:13:37.8302696Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37524 2022-11-23T02:13:37.8302959Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8303328Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8303490Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8303874Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8304047Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8304269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8304666Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8305080Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8305252Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8305632Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8305831Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8306053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8306447Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8306668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8306868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8306968Z ok (5.258s) 2022-11-23T02:13:37.8306975Z 2022-11-23T02:13:37.8307237Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8307332Z Ran 1 test in 5.258s 2022-11-23T02:13:37.8307338Z 2022-11-23T02:13:37.8307419Z OK 2022-11-23T02:13:37.8307425Z 2022-11-23T02:13:37.8307546Z Generating XML reports... 2022-11-23T02:13:37.8307984Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020603.xml 2022-11-23T02:13:37.8308294Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8308677Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8308841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8309225Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8309406Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8309412Z 2022-11-23T02:13:37.8309507Z Running tests... 2022-11-23T02:13:37.8309777Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8310109Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37730 2022-11-23T02:13:37.8310311Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37731 2022-11-23T02:13:37.8310557Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8310933Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8311094Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8311481Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8311712Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8311932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8312312Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8312464Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8312840Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8313014Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8313242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8313689Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8314088Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8314299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8314520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8314609Z ok (5.215s) 2022-11-23T02:13:37.8314615Z 2022-11-23T02:13:37.8314875Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8314978Z Ran 1 test in 5.215s 2022-11-23T02:13:37.8314983Z 2022-11-23T02:13:37.8315064Z OK 2022-11-23T02:13:37.8315070Z 2022-11-23T02:13:37.8315180Z Generating XML reports... 2022-11-23T02:13:37.8315617Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020612.xml 2022-11-23T02:13:37.8315934Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8316307Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8316469Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8316853Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8317028Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8317033Z 2022-11-23T02:13:37.8317137Z Running tests... 2022-11-23T02:13:37.8317397Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8317663Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:13:37.8317669Z 2022-11-23T02:13:37.8317929Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8318024Z Ran 1 test in 0.002s 2022-11-23T02:13:37.8318039Z 2022-11-23T02:13:37.8318123Z OK (skipped=1) 2022-11-23T02:13:37.8318129Z 2022-11-23T02:13:37.8318241Z Generating XML reports... 2022-11-23T02:13:37.8318682Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020622.xml 2022-11-23T02:13:37.8319001Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8319369Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8319532Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8319914Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8320089Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8320095Z 2022-11-23T02:13:37.8320263Z Running tests... 2022-11-23T02:13:37.8320529Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8320812Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:13:37.8320818Z 2022-11-23T02:13:37.8321081Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8321176Z Ran 1 test in 0.002s 2022-11-23T02:13:37.8321182Z 2022-11-23T02:13:37.8321276Z OK (skipped=1) 2022-11-23T02:13:37.8321281Z 2022-11-23T02:13:37.8321393Z Generating XML reports... 2022-11-23T02:13:37.8321833Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020626.xml 2022-11-23T02:13:37.8322145Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8322566Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8322733Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8323129Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8323303Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8323309Z 2022-11-23T02:13:37.8323410Z Running tests... 2022-11-23T02:13:37.8323662Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8324592Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.579s) 2022-11-23T02:13:37.8324616Z 2022-11-23T02:13:37.8324882Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8324970Z Ran 1 test in 0.579s 2022-11-23T02:13:37.8324986Z 2022-11-23T02:13:37.8325071Z OK (skipped=1) 2022-11-23T02:13:37.8325076Z 2022-11-23T02:13:37.8325195Z Generating XML reports... 2022-11-23T02:13:37.8325633Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020630.xml 2022-11-23T02:13:37.8325949Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8326315Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8326481Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8326858Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8327043Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8327051Z 2022-11-23T02:13:37.8327146Z Running tests... 2022-11-23T02:13:37.8327409Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8327794Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38135 2022-11-23T02:13:37.8328001Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38136 2022-11-23T02:13:37.8328248Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8328615Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8328784Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8329162Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8329436Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8329661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8330065Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8330432Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8330592Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8330976Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8331150Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8331360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8331822Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8332045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8332254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8332498Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp6y_p1n0 2022-11-23T02:13:37.8332746Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp6y_p1n0/_remote_module_non_scriptable.py 2022-11-23T02:13:37.8332985Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqq382onw 2022-11-23T02:13:37.8333229Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqq382onw/_remote_module_non_scriptable.py 2022-11-23T02:13:37.8333319Z ok (7.419s) 2022-11-23T02:13:37.8333325Z 2022-11-23T02:13:37.8333603Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8333703Z Ran 1 test in 7.420s 2022-11-23T02:13:37.8333709Z 2022-11-23T02:13:37.8333791Z OK 2022-11-23T02:13:37.8333797Z 2022-11-23T02:13:37.8333907Z Generating XML reports... 2022-11-23T02:13:37.8334349Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020634.xml 2022-11-23T02:13:37.8334658Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8335026Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8335193Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8335572Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8335749Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8335759Z 2022-11-23T02:13:37.8335869Z Running tests... 2022-11-23T02:13:37.8336131Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8336439Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38352 2022-11-23T02:13:37.8336634Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38353 2022-11-23T02:13:37.8336891Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8337256Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8337416Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8337801Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8337983Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8338274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8338651Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8338811Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8339189Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8339373Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8339594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8339983Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8340416Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8340642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8340854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8340942Z ok (5.610s) 2022-11-23T02:13:37.8340948Z 2022-11-23T02:13:37.8341222Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8341319Z Ran 1 test in 5.610s 2022-11-23T02:13:37.8341325Z 2022-11-23T02:13:37.8341409Z OK 2022-11-23T02:13:37.8341414Z 2022-11-23T02:13:37.8341527Z Generating XML reports... 2022-11-23T02:13:37.8341974Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020646.xml 2022-11-23T02:13:37.8342291Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8342654Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8342818Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8343198Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8343377Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8343383Z 2022-11-23T02:13:37.8343478Z Running tests... 2022-11-23T02:13:37.8343743Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8344072Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38563 2022-11-23T02:13:37.8344273Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38564 2022-11-23T02:13:37.8344532Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8344905Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8345067Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8345453Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8345634Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8345856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8346251Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8346622Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8346793Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8347238Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8347418Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8347636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8348038Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8348251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8348463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8348543Z ok (5.580s) 2022-11-23T02:13:37.8348557Z 2022-11-23T02:13:37.8348811Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8348907Z Ran 1 test in 5.581s 2022-11-23T02:13:37.8348919Z 2022-11-23T02:13:37.8349128Z OK 2022-11-23T02:13:37.8349135Z 2022-11-23T02:13:37.8349251Z Generating XML reports... 2022-11-23T02:13:37.8349692Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020656.xml 2022-11-23T02:13:37.8350013Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8350387Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8350548Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8350931Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8351106Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8351111Z 2022-11-23T02:13:37.8351208Z Running tests... 2022-11-23T02:13:37.8351474Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8352402Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.654s) 2022-11-23T02:13:37.8352409Z 2022-11-23T02:13:37.8352670Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8352768Z Ran 1 test in 0.654s 2022-11-23T02:13:37.8352774Z 2022-11-23T02:13:37.8352868Z OK (skipped=1) 2022-11-23T02:13:37.8352873Z 2022-11-23T02:13:37.8352984Z Generating XML reports... 2022-11-23T02:13:37.8353428Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020705.xml 2022-11-23T02:13:37.8353746Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8354119Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8354288Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8354670Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8354846Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8354852Z 2022-11-23T02:13:37.8354958Z Running tests... 2022-11-23T02:13:37.8355220Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8356168Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.582s) 2022-11-23T02:13:37.8356228Z 2022-11-23T02:13:37.8356523Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8356622Z Ran 1 test in 0.582s 2022-11-23T02:13:37.8356627Z 2022-11-23T02:13:37.8356720Z OK (skipped=1) 2022-11-23T02:13:37.8356726Z 2022-11-23T02:13:37.8356837Z Generating XML reports... 2022-11-23T02:13:37.8357262Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020710.xml 2022-11-23T02:13:37.8357577Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8357944Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8358113Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8358551Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8358734Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8358741Z 2022-11-23T02:13:37.8358837Z Running tests... 2022-11-23T02:13:37.8359107Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8360079Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75052 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.580s) 2022-11-23T02:13:37.8360086Z 2022-11-23T02:13:37.8360354Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8360449Z Ran 1 test in 0.580s 2022-11-23T02:13:37.8360459Z 2022-11-23T02:13:37.8360558Z OK (skipped=1) 2022-11-23T02:13:37.8360564Z 2022-11-23T02:13:37.8360678Z Generating XML reports... 2022-11-23T02:13:37.8361117Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020715.xml 2022-11-23T02:13:37.8361428Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8361804Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8361966Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8362349Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8362522Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8362528Z 2022-11-23T02:13:37.8362624Z Running tests... 2022-11-23T02:13:37.8362888Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8363896Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75139 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.606s) 2022-11-23T02:13:37.8363903Z 2022-11-23T02:13:37.8364167Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8364263Z Ran 1 test in 0.606s 2022-11-23T02:13:37.8364269Z 2022-11-23T02:13:37.8364362Z OK (skipped=1) 2022-11-23T02:13:37.8364367Z 2022-11-23T02:13:37.8364478Z Generating XML reports... 2022-11-23T02:13:37.8364924Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020719.xml 2022-11-23T02:13:37.8365296Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8365674Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8365833Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8366203Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8366377Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8366399Z 2022-11-23T02:13:37.8366487Z Running tests... 2022-11-23T02:13:37.8366752Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8367797Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84886 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.581s) 2022-11-23T02:13:37.8367835Z 2022-11-23T02:13:37.8368096Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8368193Z Ran 1 test in 0.582s 2022-11-23T02:13:37.8368199Z 2022-11-23T02:13:37.8368294Z OK (skipped=1) 2022-11-23T02:13:37.8368300Z 2022-11-23T02:13:37.8368414Z Generating XML reports... 2022-11-23T02:13:37.8368869Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020724.xml 2022-11-23T02:13:37.8369175Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8369557Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8369720Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8370118Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8370294Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8370300Z 2022-11-23T02:13:37.8370396Z Running tests... 2022-11-23T02:13:37.8370681Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8370986Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39104 2022-11-23T02:13:37.8371190Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39105 2022-11-23T02:13:37.8371443Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8371836Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8371999Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8372393Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8372567Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8372792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8373180Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8373341Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8373715Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8373903Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8374125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8374590Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8375006Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8375233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8375445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8375662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:13:37.8375884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:13:37.8376277Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.8376726Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.8377087Z STAGE:2022-11-23 02:07:32 39104:39104 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8377415Z STAGE:2022-11-23 02:07:32 39105:39105 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8377778Z STAGE:2022-11-23 02:07:32 39105:39105 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8378126Z STAGE:2022-11-23 02:07:32 39105:39105 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8378459Z STAGE:2022-11-23 02:07:32 39104:39104 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8378822Z STAGE:2022-11-23 02:07:32 39104:39104 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8379152Z STAGE:2022-11-23 02:07:32 39105:39105 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8379497Z STAGE:2022-11-23 02:07:32 39104:39104 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8379827Z STAGE:2022-11-23 02:07:32 39105:39105 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8380189Z STAGE:2022-11-23 02:07:32 39105:39105 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8380519Z STAGE:2022-11-23 02:07:32 39104:39104 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8380883Z STAGE:2022-11-23 02:07:32 39104:39104 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8380973Z ok (5.010s) 2022-11-23T02:13:37.8380981Z 2022-11-23T02:13:37.8381243Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8381331Z Ran 1 test in 5.010s 2022-11-23T02:13:37.8381345Z 2022-11-23T02:13:37.8381418Z OK 2022-11-23T02:13:37.8381423Z 2022-11-23T02:13:37.8381534Z Generating XML reports... 2022-11-23T02:13:37.8381991Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020729.xml 2022-11-23T02:13:37.8382300Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8382693Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8382855Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8383246Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8383422Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8383428Z 2022-11-23T02:13:37.8383524Z Running tests... 2022-11-23T02:13:37.8383789Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8384119Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39323 2022-11-23T02:13:37.8384388Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39324 2022-11-23T02:13:37.8384641Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8385037Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8385198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8385579Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8385766Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8385986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8386400Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8386583Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8386963Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8387134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8387344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8387738Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8388141Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8388351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8388585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8388804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:13:37.8389211Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.8389429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:13:37.8389819Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.8390164Z STAGE:2022-11-23 02:07:41 39323:39323 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8390493Z STAGE:2022-11-23 02:07:41 39324:39324 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8390827Z STAGE:2022-11-23 02:07:41 39323:39323 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8391202Z STAGE:2022-11-23 02:07:41 39323:39323 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8391537Z STAGE:2022-11-23 02:07:41 39324:39324 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8391882Z STAGE:2022-11-23 02:07:41 39324:39324 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8392209Z STAGE:2022-11-23 02:07:41 39323:39323 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8392546Z STAGE:2022-11-23 02:07:41 39324:39324 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8392878Z STAGE:2022-11-23 02:07:41 39323:39323 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8393223Z STAGE:2022-11-23 02:07:41 39323:39323 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8393580Z STAGE:2022-11-23 02:07:41 39324:39324 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8393987Z STAGE:2022-11-23 02:07:41 39324:39324 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8394079Z ok (5.510s) 2022-11-23T02:13:37.8394085Z 2022-11-23T02:13:37.8394356Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8394452Z Ran 1 test in 5.511s 2022-11-23T02:13:37.8394458Z 2022-11-23T02:13:37.8394530Z OK 2022-11-23T02:13:37.8394544Z 2022-11-23T02:13:37.8394648Z Generating XML reports... 2022-11-23T02:13:37.8395081Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020738.xml 2022-11-23T02:13:37.8395402Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8395772Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8395997Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8396386Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8396562Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8396569Z 2022-11-23T02:13:37.8396674Z Running tests... 2022-11-23T02:13:37.8396935Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8397242Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39542 2022-11-23T02:13:37.8397446Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39543 2022-11-23T02:13:37.8397704Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8398075Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8398237Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8398625Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8398801Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8399024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8399416Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8399792Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8399953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8400328Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8400509Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8400739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8401125Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8401336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8401544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8401772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:13:37.8402159Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.8402382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:13:37.8402856Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.8403179Z STAGE:2022-11-23 02:07:50 39542:39542 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8403513Z STAGE:2022-11-23 02:07:50 39543:39543 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8403846Z STAGE:2022-11-23 02:07:50 39542:39542 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8404200Z STAGE:2022-11-23 02:07:50 39542:39542 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8404533Z STAGE:2022-11-23 02:07:50 39543:39543 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8404880Z STAGE:2022-11-23 02:07:50 39543:39543 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8405257Z STAGE:2022-11-23 02:07:50 39542:39542 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8405595Z STAGE:2022-11-23 02:07:50 39543:39543 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8405929Z STAGE:2022-11-23 02:07:50 39542:39542 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8406273Z STAGE:2022-11-23 02:07:50 39542:39542 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8406609Z STAGE:2022-11-23 02:07:50 39543:39543 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8406954Z STAGE:2022-11-23 02:07:50 39543:39543 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8407044Z ok (5.611s) 2022-11-23T02:13:37.8407050Z 2022-11-23T02:13:37.8407340Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8407440Z Ran 1 test in 5.612s 2022-11-23T02:13:37.8407446Z 2022-11-23T02:13:37.8407532Z OK 2022-11-23T02:13:37.8407544Z 2022-11-23T02:13:37.8407656Z Generating XML reports... 2022-11-23T02:13:37.8408197Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020748.xml 2022-11-23T02:13:37.8408508Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8408884Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8409046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8409431Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8409604Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8409610Z 2022-11-23T02:13:37.8409706Z Running tests... 2022-11-23T02:13:37.8409970Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8410285Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39761 2022-11-23T02:13:37.8410489Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39762 2022-11-23T02:13:37.8410740Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8411113Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8411274Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8411657Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8411833Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8412059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8412528Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8412890Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8413056Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8413435Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8413612Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8413842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8414234Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8414497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8414720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8414932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:13:37.8415149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:13:37.8415541Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.8415935Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.8416263Z STAGE:2022-11-23 02:08:00 39761:39761 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8416594Z STAGE:2022-11-23 02:08:00 39762:39762 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8416931Z STAGE:2022-11-23 02:08:00 39761:39761 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8417285Z STAGE:2022-11-23 02:08:00 39761:39761 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8417616Z STAGE:2022-11-23 02:08:00 39762:39762 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8417969Z STAGE:2022-11-23 02:08:00 39762:39762 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8418293Z STAGE:2022-11-23 02:08:00 39761:39761 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8418616Z STAGE:2022-11-23 02:08:00 39762:39762 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8418954Z STAGE:2022-11-23 02:08:00 39761:39761 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8419301Z STAGE:2022-11-23 02:08:00 39761:39761 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8419637Z STAGE:2022-11-23 02:08:00 39762:39762 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8419985Z STAGE:2022-11-23 02:08:00 39762:39762 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8420075Z ok (5.020s) 2022-11-23T02:13:37.8420081Z 2022-11-23T02:13:37.8420345Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8420442Z Ran 1 test in 5.020s 2022-11-23T02:13:37.8420448Z 2022-11-23T02:13:37.8420538Z OK 2022-11-23T02:13:37.8420544Z 2022-11-23T02:13:37.8420655Z Generating XML reports... 2022-11-23T02:13:37.8421091Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020757.xml 2022-11-23T02:13:37.8421413Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8421787Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8421999Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8422380Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8422555Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8422561Z 2022-11-23T02:13:37.8422660Z Running tests... 2022-11-23T02:13:37.8422931Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8423226Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39980 2022-11-23T02:13:37.8423435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39981 2022-11-23T02:13:37.8423683Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8424104Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8424277Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8424665Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8424843Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8425067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8425437Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8425607Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8425984Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8426163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8426391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8426803Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8427190Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8427405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8427624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8427769Z skip: Skipped due to small world size. (5.225s) 2022-11-23T02:13:37.8427776Z 2022-11-23T02:13:37.8428037Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8428125Z Ran 1 test in 5.225s 2022-11-23T02:13:37.8428130Z 2022-11-23T02:13:37.8428230Z OK (skipped=1) 2022-11-23T02:13:37.8428240Z 2022-11-23T02:13:37.8428353Z Generating XML reports... 2022-11-23T02:13:37.8428799Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020806.xml 2022-11-23T02:13:37.8429110Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8429485Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8429647Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8430028Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8430216Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8430222Z 2022-11-23T02:13:37.8430319Z Running tests... 2022-11-23T02:13:37.8430583Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8430938Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40187 2022-11-23T02:13:37.8431187Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40188 2022-11-23T02:13:37.8431483Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8431933Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8432123Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8432567Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8432780Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8433095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8433546Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8433736Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8434184Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8434380Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8434640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8435104Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8435570Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8435822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8436078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8436252Z skip: Skipped due to small world size. (4.824s) 2022-11-23T02:13:37.8436260Z 2022-11-23T02:13:37.8436569Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8436694Z Ran 1 test in 4.824s 2022-11-23T02:13:37.8436701Z 2022-11-23T02:13:37.8436810Z OK (skipped=1) 2022-11-23T02:13:37.8436816Z 2022-11-23T02:13:37.8436948Z Generating XML reports... 2022-11-23T02:13:37.8437476Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020816.xml 2022-11-23T02:13:37.8437842Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8438275Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8438478Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8438931Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8439139Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8439145Z 2022-11-23T02:13:37.8439258Z Running tests... 2022-11-23T02:13:37.8439577Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8439936Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40394 2022-11-23T02:13:37.8440177Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40395 2022-11-23T02:13:37.8440471Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8440912Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8441195Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8441640Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8441852Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8442113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8442546Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8442737Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8443222Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8443431Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8443765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8444235Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8444693Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8444945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8445189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8445361Z skip: Skipped due to small world size. (4.813s) 2022-11-23T02:13:37.8445368Z 2022-11-23T02:13:37.8445677Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8445790Z Ran 1 test in 4.813s 2022-11-23T02:13:37.8445797Z 2022-11-23T02:13:37.8445909Z OK (skipped=1) 2022-11-23T02:13:37.8445920Z 2022-11-23T02:13:37.8446055Z Generating XML reports... 2022-11-23T02:13:37.8446572Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020825.xml 2022-11-23T02:13:37.8446937Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8447371Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8447561Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8448075Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8448295Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8448303Z 2022-11-23T02:13:37.8448404Z Running tests... 2022-11-23T02:13:37.8448669Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8448972Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40601 2022-11-23T02:13:37.8449175Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40602 2022-11-23T02:13:37.8449432Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8449807Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8449968Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8450349Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8450527Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8450751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8451188Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8451349Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8451727Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8451919Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8452155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8452553Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8452952Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8453227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8453451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8453595Z skip: Skipped due to small world size. (4.820s) 2022-11-23T02:13:37.8453601Z 2022-11-23T02:13:37.8453869Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8453978Z Ran 1 test in 4.821s 2022-11-23T02:13:37.8453984Z 2022-11-23T02:13:37.8454088Z OK (skipped=1) 2022-11-23T02:13:37.8454094Z 2022-11-23T02:13:37.8454204Z Generating XML reports... 2022-11-23T02:13:37.8454662Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020834.xml 2022-11-23T02:13:37.8454975Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8455343Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8455510Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8455961Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8456202Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8456209Z 2022-11-23T02:13:37.8456331Z Running tests... 2022-11-23T02:13:37.8456653Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8456995Z test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40808 2022-11-23T02:13:37.8457235Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40809 2022-11-23T02:13:37.8457528Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8457965Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8458160Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8458604Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8458813Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8459080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8459544Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8459974Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8460161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8460614Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8460891Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8461141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8461604Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8461850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8462234Z STAGE:2022-11-23 02:08:45 40809:40809 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8462479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8462867Z STAGE:2022-11-23 02:08:45 40808:40808 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8463262Z STAGE:2022-11-23 02:08:45 40808:40808 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8463713Z STAGE:2022-11-23 02:08:45 40809:40809 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8464135Z STAGE:2022-11-23 02:08:45 40808:40808 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8464542Z STAGE:2022-11-23 02:08:45 40809:40809 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8464931Z STAGE:2022-11-23 02:08:45 40809:40809 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8465313Z STAGE:2022-11-23 02:08:45 40808:40808 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8465703Z STAGE:2022-11-23 02:08:45 40809:40809 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8466096Z STAGE:2022-11-23 02:08:45 40808:40808 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8466508Z STAGE:2022-11-23 02:08:45 40809:40809 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8466920Z STAGE:2022-11-23 02:08:45 40808:40808 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8467025Z ok (5.312s) 2022-11-23T02:13:37.8467033Z 2022-11-23T02:13:37.8467341Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8467455Z Ran 1 test in 5.312s 2022-11-23T02:13:37.8467462Z 2022-11-23T02:13:37.8467560Z OK 2022-11-23T02:13:37.8467566Z 2022-11-23T02:13:37.8467697Z Generating XML reports... 2022-11-23T02:13:37.8468226Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020843.xml 2022-11-23T02:13:37.8468551Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8468908Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8469073Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8469456Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8469629Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8469635Z 2022-11-23T02:13:37.8469732Z Running tests... 2022-11-23T02:13:37.8469995Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8470283Z test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41021 2022-11-23T02:13:37.8470487Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41022 2022-11-23T02:13:37.8470741Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8471110Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8471333Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8471713Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8471889Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8472113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8472505Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8472870Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8473033Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8473410Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8473646Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8473867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8474258Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8474471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8474798Z STAGE:2022-11-23 02:08:55 41022:41022 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8475001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8475324Z STAGE:2022-11-23 02:08:55 41021:41021 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8475659Z STAGE:2022-11-23 02:08:55 41021:41021 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8476011Z STAGE:2022-11-23 02:08:55 41021:41021 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8476343Z STAGE:2022-11-23 02:08:55 41022:41022 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8476690Z STAGE:2022-11-23 02:08:55 41022:41022 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8477017Z STAGE:2022-11-23 02:08:55 41022:41022 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8477341Z STAGE:2022-11-23 02:08:55 41021:41021 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8477674Z STAGE:2022-11-23 02:08:55 41021:41021 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8478018Z STAGE:2022-11-23 02:08:55 41021:41021 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8478350Z STAGE:2022-11-23 02:08:55 41022:41022 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8478706Z STAGE:2022-11-23 02:08:55 41022:41022 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8478796Z ok (5.151s) 2022-11-23T02:13:37.8478802Z 2022-11-23T02:13:37.8479064Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8479165Z Ran 1 test in 5.152s 2022-11-23T02:13:37.8479171Z 2022-11-23T02:13:37.8479252Z OK 2022-11-23T02:13:37.8479258Z 2022-11-23T02:13:37.8479372Z Generating XML reports... 2022-11-23T02:13:37.8479808Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020852.xml 2022-11-23T02:13:37.8480116Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8480487Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8480649Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8481091Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8481265Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8481272Z 2022-11-23T02:13:37.8481360Z Running tests... 2022-11-23T02:13:37.8481624Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8481884Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s) 2022-11-23T02:13:37.8481890Z 2022-11-23T02:13:37.8482151Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8482249Z Ran 1 test in 0.002s 2022-11-23T02:13:37.8482255Z 2022-11-23T02:13:37.8482349Z OK (skipped=1) 2022-11-23T02:13:37.8482354Z 2022-11-23T02:13:37.8482465Z Generating XML reports... 2022-11-23T02:13:37.8482949Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020901.xml 2022-11-23T02:13:37.8483268Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8483637Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8483798Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8484174Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8484350Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8484356Z 2022-11-23T02:13:37.8484451Z Running tests... 2022-11-23T02:13:37.8484713Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8485008Z test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41300 2022-11-23T02:13:37.8485217Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41301 2022-11-23T02:13:37.8485466Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8485836Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8485997Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8486377Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8486550Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8486761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8487128Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8487297Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8487673Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8487958Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8488178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8488579Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8488970Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8489182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8489515Z STAGE:2022-11-23 02:09:08 41301:41301 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8489800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8490129Z STAGE:2022-11-23 02:09:08 41300:41300 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8490459Z STAGE:2022-11-23 02:09:08 41301:41301 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8490788Z STAGE:2022-11-23 02:09:08 41300:41300 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8491135Z STAGE:2022-11-23 02:09:08 41301:41301 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8491481Z STAGE:2022-11-23 02:09:08 41300:41300 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8491806Z STAGE:2022-11-23 02:09:08 41301:41301 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8492130Z STAGE:2022-11-23 02:09:08 41300:41300 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8492514Z STAGE:2022-11-23 02:09:08 41300:41300 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8492866Z STAGE:2022-11-23 02:09:08 41300:41300 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8493196Z STAGE:2022-11-23 02:09:08 41301:41301 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8493542Z STAGE:2022-11-23 02:09:08 41301:41301 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8493633Z ok (5.044s) 2022-11-23T02:13:37.8493639Z 2022-11-23T02:13:37.8493905Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8493995Z Ran 1 test in 5.044s 2022-11-23T02:13:37.8494009Z 2022-11-23T02:13:37.8494083Z OK 2022-11-23T02:13:37.8494088Z 2022-11-23T02:13:37.8494199Z Generating XML reports... 2022-11-23T02:13:37.8494636Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020906.xml 2022-11-23T02:13:37.8494949Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8495316Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8495477Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8495853Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8496030Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8496036Z 2022-11-23T02:13:37.8496134Z Running tests... 2022-11-23T02:13:37.8496395Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8496673Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce_scatter_tensor (0.002s) 2022-11-23T02:13:37.8496680Z 2022-11-23T02:13:37.8496953Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8497050Z Ran 1 test in 0.002s 2022-11-23T02:13:37.8497055Z 2022-11-23T02:13:37.8497149Z OK (skipped=1) 2022-11-23T02:13:37.8497154Z 2022-11-23T02:13:37.8497265Z Generating XML reports... 2022-11-23T02:13:37.8497702Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020915.xml 2022-11-23T02:13:37.8498008Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8498376Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8498536Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8498912Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8499091Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8499153Z 2022-11-23T02:13:37.8499250Z Running tests... 2022-11-23T02:13:37.8499508Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8499763Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports reduce_scatter_v (0.003s) 2022-11-23T02:13:37.8499769Z 2022-11-23T02:13:37.8500030Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8500128Z Ran 1 test in 0.003s 2022-11-23T02:13:37.8500134Z 2022-11-23T02:13:37.8500227Z OK (skipped=1) 2022-11-23T02:13:37.8500233Z 2022-11-23T02:13:37.8500344Z Generating XML reports... 2022-11-23T02:13:37.8500777Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020919.xml 2022-11-23T02:13:37.8501086Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8501501Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8501671Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8502052Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8502227Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8502233Z 2022-11-23T02:13:37.8502329Z Running tests... 2022-11-23T02:13:37.8502591Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8502878Z test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41645 2022-11-23T02:13:37.8503083Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41646 2022-11-23T02:13:37.8503337Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8503710Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8503871Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8504251Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8504426Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8504643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8505001Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8505162Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8505541Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8505722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8505945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8506340Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8506726Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8506938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8507266Z STAGE:2022-11-23 02:09:25 41646:41646 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8507482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8507809Z STAGE:2022-11-23 02:09:26 41645:41645 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8508205Z STAGE:2022-11-23 02:09:26 41646:41646 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8508553Z STAGE:2022-11-23 02:09:26 41646:41646 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8508887Z STAGE:2022-11-23 02:09:26 41645:41645 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8509234Z STAGE:2022-11-23 02:09:26 41645:41645 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8509561Z STAGE:2022-11-23 02:09:26 41646:41646 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8509889Z STAGE:2022-11-23 02:09:26 41645:41645 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8510220Z STAGE:2022-11-23 02:09:26 41645:41645 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8510611Z STAGE:2022-11-23 02:09:26 41645:41645 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8510952Z STAGE:2022-11-23 02:09:26 41646:41646 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8511295Z STAGE:2022-11-23 02:09:26 41646:41646 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8511386Z ok (5.213s) 2022-11-23T02:13:37.8511393Z 2022-11-23T02:13:37.8511656Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8511744Z Ran 1 test in 5.214s 2022-11-23T02:13:37.8511760Z 2022-11-23T02:13:37.8511832Z OK 2022-11-23T02:13:37.8511838Z 2022-11-23T02:13:37.8511950Z Generating XML reports... 2022-11-23T02:13:37.8512392Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020923.xml 2022-11-23T02:13:37.8512700Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8513073Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8513235Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8513614Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8513790Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8513797Z 2022-11-23T02:13:37.8513894Z Running tests... 2022-11-23T02:13:37.8514157Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8514398Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s) 2022-11-23T02:13:37.8514404Z 2022-11-23T02:13:37.8514666Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8514764Z Ran 1 test in 0.002s 2022-11-23T02:13:37.8514770Z 2022-11-23T02:13:37.8514869Z OK (skipped=1) 2022-11-23T02:13:37.8514874Z 2022-11-23T02:13:37.8514999Z Generating XML reports... 2022-11-23T02:13:37.8515432Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020932.xml 2022-11-23T02:13:37.8515744Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8516111Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8516275Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8516653Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8516827Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8516833Z 2022-11-23T02:13:37.8516928Z Running tests... 2022-11-23T02:13:37.8517182Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8517432Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s) 2022-11-23T02:13:37.8517491Z 2022-11-23T02:13:37.8517755Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8517851Z Ran 1 test in 0.002s 2022-11-23T02:13:37.8517857Z 2022-11-23T02:13:37.8517951Z OK (skipped=1) 2022-11-23T02:13:37.8517957Z 2022-11-23T02:13:37.8518069Z Generating XML reports... 2022-11-23T02:13:37.8518505Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020936.xml 2022-11-23T02:13:37.8518814Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8519181Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8519345Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8519772Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8519951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8519957Z 2022-11-23T02:13:37.8520053Z Running tests... 2022-11-23T02:13:37.8520319Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8520614Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41990 2022-11-23T02:13:37.8520821Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41991 2022-11-23T02:13:37.8521076Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8521444Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8521606Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8521994Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8522167Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8522387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8522769Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8523134Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8523295Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8523673Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8523847Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8524074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8524463Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8524676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8525005Z STAGE:2022-11-23 02:09:43 41991:41991 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8525216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8525544Z STAGE:2022-11-23 02:09:43 41990:41990 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8525876Z STAGE:2022-11-23 02:09:43 41990:41990 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8526222Z STAGE:2022-11-23 02:09:43 41990:41990 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8526619Z STAGE:2022-11-23 02:09:43 41991:41991 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8526965Z STAGE:2022-11-23 02:09:43 41991:41991 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8527293Z STAGE:2022-11-23 02:09:43 41990:41990 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8527616Z STAGE:2022-11-23 02:09:43 41991:41991 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8527997Z STAGE:2022-11-23 02:09:43 41990:41990 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8528344Z STAGE:2022-11-23 02:09:43 41990:41990 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8528674Z STAGE:2022-11-23 02:09:43 41991:41991 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8529074Z STAGE:2022-11-23 02:09:43 41991:41991 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8529170Z ok (5.110s) 2022-11-23T02:13:37.8529176Z 2022-11-23T02:13:37.8529441Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8529529Z Ran 1 test in 5.111s 2022-11-23T02:13:37.8529544Z 2022-11-23T02:13:37.8529617Z OK 2022-11-23T02:13:37.8529623Z 2022-11-23T02:13:37.8529737Z Generating XML reports... 2022-11-23T02:13:37.8530174Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020940.xml 2022-11-23T02:13:37.8530483Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8530852Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8531015Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8531399Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8531575Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8531581Z 2022-11-23T02:13:37.8531680Z Running tests... 2022-11-23T02:13:37.8531943Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8532226Z test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42203 2022-11-23T02:13:37.8532429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42204 2022-11-23T02:13:37.8532680Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8533046Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8533208Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8533592Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8533764Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8533987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8534383Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8534747Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8534906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8535283Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8535448Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8535732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8536127Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8536339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8536669Z STAGE:2022-11-23 02:09:52 42204:42204 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8536881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8537206Z STAGE:2022-11-23 02:09:52 42203:42203 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8537537Z STAGE:2022-11-23 02:09:52 42203:42203 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8537881Z STAGE:2022-11-23 02:09:52 42203:42203 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8538267Z STAGE:2022-11-23 02:09:52 42204:42204 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8538617Z STAGE:2022-11-23 02:09:52 42204:42204 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8538943Z STAGE:2022-11-23 02:09:52 42203:42203 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8539267Z STAGE:2022-11-23 02:09:52 42204:42204 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8539597Z STAGE:2022-11-23 02:09:52 42203:42203 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8540159Z STAGE:2022-11-23 02:09:52 42203:42203 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 02:09:52 42204:42204 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8540166Z 2022-11-23T02:13:37.8540512Z STAGE:2022-11-23 02:09:52 42204:42204 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8540609Z ok (5.018s) 2022-11-23T02:13:37.8540615Z 2022-11-23T02:13:37.8540877Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8540974Z Ran 1 test in 5.019s 2022-11-23T02:13:37.8540980Z 2022-11-23T02:13:37.8541060Z OK 2022-11-23T02:13:37.8541065Z 2022-11-23T02:13:37.8541181Z Generating XML reports... 2022-11-23T02:13:37.8541616Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020949.xml 2022-11-23T02:13:37.8541927Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8542301Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8542461Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8542834Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8543011Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8543017Z 2022-11-23T02:13:37.8543112Z Running tests... 2022-11-23T02:13:37.8543376Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8543677Z test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42416 2022-11-23T02:13:37.8543882Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42417 2022-11-23T02:13:37.8544133Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8544501Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8544663Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8545043Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8545289Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8545512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8545882Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8546045Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8546422Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8546595Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8546818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8547257Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8547658Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8547870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8548083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8548174Z ok (4.909s) 2022-11-23T02:13:37.8548180Z 2022-11-23T02:13:37.8548434Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8548531Z Ran 1 test in 4.910s 2022-11-23T02:13:37.8548537Z 2022-11-23T02:13:37.8548618Z OK 2022-11-23T02:13:37.8548624Z 2022-11-23T02:13:37.8548736Z Generating XML reports... 2022-11-23T02:13:37.8549178Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020959.xml 2022-11-23T02:13:37.8549495Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8549863Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8550026Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8550407Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8550584Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8550590Z 2022-11-23T02:13:37.8550685Z Running tests... 2022-11-23T02:13:37.8550951Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8551250Z test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42623 2022-11-23T02:13:37.8551450Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42624 2022-11-23T02:13:37.8551706Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8552074Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8552234Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8552611Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8552786Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8553012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8553377Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8553537Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8553966Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8554143Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8554364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8554755Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8555140Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8555350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8555681Z STAGE:2022-11-23 02:10:10 42624:42624 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8555892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8556274Z STAGE:2022-11-23 02:10:10 42623:42623 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8556613Z STAGE:2022-11-23 02:10:10 42623:42623 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8556957Z STAGE:2022-11-23 02:10:10 42623:42623 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8557291Z STAGE:2022-11-23 02:10:10 42624:42624 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8557639Z STAGE:2022-11-23 02:10:10 42624:42624 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8557964Z STAGE:2022-11-23 02:10:10 42623:42623 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8558286Z STAGE:2022-11-23 02:10:10 42624:42624 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8558623Z STAGE:2022-11-23 02:10:10 42623:42623 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8558971Z STAGE:2022-11-23 02:10:10 42623:42623 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8559302Z STAGE:2022-11-23 02:10:10 42624:42624 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8559646Z STAGE:2022-11-23 02:10:10 42624:42624 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8559736Z ok (5.215s) 2022-11-23T02:13:37.8559742Z 2022-11-23T02:13:37.8560006Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8560104Z Ran 1 test in 5.215s 2022-11-23T02:13:37.8560110Z 2022-11-23T02:13:37.8560194Z OK 2022-11-23T02:13:37.8560200Z 2022-11-23T02:13:37.8560303Z Generating XML reports... 2022-11-23T02:13:37.8560736Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021008.xml 2022-11-23T02:13:37.8561052Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8561423Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8561583Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8561962Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8562135Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8562141Z 2022-11-23T02:13:37.8562237Z Running tests... 2022-11-23T02:13:37.8562498Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8562733Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s) 2022-11-23T02:13:37.8562739Z 2022-11-23T02:13:37.8562999Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8563149Z Ran 1 test in 0.002s 2022-11-23T02:13:37.8563158Z 2022-11-23T02:13:37.8563253Z OK (skipped=1) 2022-11-23T02:13:37.8563258Z 2022-11-23T02:13:37.8563372Z Generating XML reports... 2022-11-23T02:13:37.8563810Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021017.xml 2022-11-23T02:13:37.8564121Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8564489Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8564649Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8565025Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8565200Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8565206Z 2022-11-23T02:13:37.8565302Z Running tests... 2022-11-23T02:13:37.8565617Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8565857Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s) 2022-11-23T02:13:37.8565873Z 2022-11-23T02:13:37.8566126Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8566223Z Ran 1 test in 0.002s 2022-11-23T02:13:37.8566228Z 2022-11-23T02:13:37.8566324Z OK (skipped=1) 2022-11-23T02:13:37.8566329Z 2022-11-23T02:13:37.8566442Z Generating XML reports... 2022-11-23T02:13:37.8566880Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021021.xml 2022-11-23T02:13:37.8567189Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8567558Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8567777Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8568156Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8568330Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8568335Z 2022-11-23T02:13:37.8568432Z Running tests... 2022-11-23T02:13:37.8568694Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8568996Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42968 2022-11-23T02:13:37.8569199Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42969 2022-11-23T02:13:37.8569448Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8569820Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8569984Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8570364Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8570537Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8570759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8571125Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8571285Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8571655Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8571829Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8572118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8572513Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8572905Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8573117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8573329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8573549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:13:37.8573767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:13:37.8574211Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.8574608Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:13:37.8574934Z STAGE:2022-11-23 02:10:28 42968:42968 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8575261Z STAGE:2022-11-23 02:10:28 42969:42969 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8575813Z STAGE:2022-11-23 02:10:28 42968:42968 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:10:28 42969:42969 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8575820Z 2022-11-23T02:13:37.8576395Z STAGE:2022-11-23 02:10:28 42969:42969 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 02:10:28 42968:42968 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8576402Z 2022-11-23T02:13:37.8576733Z STAGE:2022-11-23 02:10:28 42968:42968 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8577055Z STAGE:2022-11-23 02:10:28 42969:42969 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8577386Z STAGE:2022-11-23 02:10:28 42968:42968 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8577714Z STAGE:2022-11-23 02:10:28 42969:42969 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8578059Z STAGE:2022-11-23 02:10:28 42968:42968 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8578405Z STAGE:2022-11-23 02:10:28 42969:42969 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8578495Z ok (4.909s) 2022-11-23T02:13:37.8578501Z 2022-11-23T02:13:37.8578766Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8578863Z Ran 1 test in 4.909s 2022-11-23T02:13:37.8578869Z 2022-11-23T02:13:37.8578953Z OK 2022-11-23T02:13:37.8578964Z 2022-11-23T02:13:37.8579076Z Generating XML reports... 2022-11-23T02:13:37.8579514Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021025.xml 2022-11-23T02:13:37.8579827Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8580190Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8580350Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8580729Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8580903Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8580909Z 2022-11-23T02:13:37.8581006Z Running tests... 2022-11-23T02:13:37.8581267Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8581621Z test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43187 2022-11-23T02:13:37.8581825Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43188 2022-11-23T02:13:37.8582076Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8582446Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8582607Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8582988Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8583160Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8583430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8583806Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8583967Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8584348Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8584523Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8584745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8585138Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8585531Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8585749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8585961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8586098Z skip: Skipped due to small world size. (5.179s) 2022-11-23T02:13:37.8586114Z 2022-11-23T02:13:37.8586367Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8586463Z Ran 1 test in 5.180s 2022-11-23T02:13:37.8586471Z 2022-11-23T02:13:37.8586564Z OK (skipped=1) 2022-11-23T02:13:37.8586570Z 2022-11-23T02:13:37.8586682Z Generating XML reports... 2022-11-23T02:13:37.8587120Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021034.xml 2022-11-23T02:13:37.8587427Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8587796Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8587970Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8588349Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8588523Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8588529Z 2022-11-23T02:13:37.8588626Z Running tests... 2022-11-23T02:13:37.8588888Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8589190Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43394 2022-11-23T02:13:37.8589398Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43395 2022-11-23T02:13:37.8589647Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8590021Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8590234Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8590614Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8590789Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8591010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8591374Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8591528Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8591903Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8592077Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8592353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8592753Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8593139Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8593352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8593564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8593652Z ok (4.940s) 2022-11-23T02:13:37.8593658Z 2022-11-23T02:13:37.8593919Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8594016Z Ran 1 test in 4.940s 2022-11-23T02:13:37.8594022Z 2022-11-23T02:13:37.8594104Z OK 2022-11-23T02:13:37.8594110Z 2022-11-23T02:13:37.8594226Z Generating XML reports... 2022-11-23T02:13:37.8594665Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021043.xml 2022-11-23T02:13:37.8594973Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8595342Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8595504Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8595880Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8596054Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8596060Z 2022-11-23T02:13:37.8596157Z Running tests... 2022-11-23T02:13:37.8596419Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8596708Z test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43601 2022-11-23T02:13:37.8596907Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43602 2022-11-23T02:13:37.8597155Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8597525Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8597688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8598067Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8598244Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8598469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8598840Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8599060Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8599438Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8599611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8599833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8600224Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8600613Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8600824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8601094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8601187Z ok (5.124s) 2022-11-23T02:13:37.8601193Z 2022-11-23T02:13:37.8601461Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8601558Z Ran 1 test in 5.124s 2022-11-23T02:13:37.8601564Z 2022-11-23T02:13:37.8601646Z OK 2022-11-23T02:13:37.8601652Z 2022-11-23T02:13:37.8601762Z Generating XML reports... 2022-11-23T02:13:37.8602198Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021052.xml 2022-11-23T02:13:37.8602504Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8602864Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8603025Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8603406Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8603582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8603588Z 2022-11-23T02:13:37.8603686Z Running tests... 2022-11-23T02:13:37.8603947Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8604248Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43808 2022-11-23T02:13:37.8604448Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43809 2022-11-23T02:13:37.8604697Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8605065Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8605230Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8605612Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8605785Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8606008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8606374Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8606538Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8606916Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8607091Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8607312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8607879Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8608273Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8608485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8608697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8608778Z ok (5.012s) 2022-11-23T02:13:37.8608783Z 2022-11-23T02:13:37.8609047Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8609143Z Ran 1 test in 5.013s 2022-11-23T02:13:37.8609149Z 2022-11-23T02:13:37.8609230Z OK 2022-11-23T02:13:37.8609236Z 2022-11-23T02:13:37.8609347Z Generating XML reports... 2022-11-23T02:13:37.8609867Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021102.xml 2022-11-23T02:13:37.8610187Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8610554Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8610715Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8611092Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8611265Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8611273Z 2022-11-23T02:13:37.8611368Z Running tests... 2022-11-23T02:13:37.8611629Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8611954Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44015 2022-11-23T02:13:37.8612164Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44016 2022-11-23T02:13:37.8612409Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8612775Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8612936Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8613313Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8613486Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8613708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8614075Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8614236Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8614612Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8614786Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8615006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8615398Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8615787Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8616000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8616327Z STAGE:2022-11-23 02:11:14 44016:44016 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8616603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8616935Z STAGE:2022-11-23 02:11:14 44015:44015 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8617266Z STAGE:2022-11-23 02:11:14 44015:44015 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8617614Z STAGE:2022-11-23 02:11:14 44015:44015 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8617950Z STAGE:2022-11-23 02:11:14 44016:44016 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8618294Z STAGE:2022-11-23 02:11:14 44016:44016 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8618385Z ok (4.929s) 2022-11-23T02:13:37.8618390Z 2022-11-23T02:13:37.8618652Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8618749Z Ran 1 test in 4.930s 2022-11-23T02:13:37.8618759Z 2022-11-23T02:13:37.8618914Z OK 2022-11-23T02:13:37.8618921Z 2022-11-23T02:13:37.8619035Z Generating XML reports... 2022-11-23T02:13:37.8619476Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021111.xml 2022-11-23T02:13:37.8619784Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8620155Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8620309Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8620687Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8620863Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8620871Z 2022-11-23T02:13:37.8620967Z Running tests... 2022-11-23T02:13:37.8621232Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8621559Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44228 2022-11-23T02:13:37.8621762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44229 2022-11-23T02:13:37.8622013Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8622381Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8622545Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8622921Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8623098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8623322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8623692Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8623853Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8624229Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8624403Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8624627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8625018Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8625404Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8625672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8626005Z STAGE:2022-11-23 02:11:23 44229:44229 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8626216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8626533Z STAGE:2022-11-23 02:11:23 44228:44228 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8626868Z STAGE:2022-11-23 02:11:23 44229:44229 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8627215Z STAGE:2022-11-23 02:11:23 44229:44229 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8627547Z STAGE:2022-11-23 02:11:23 44228:44228 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8627892Z STAGE:2022-11-23 02:11:23 44228:44228 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8628032Z ok (5.678s) 2022-11-23T02:13:37.8628040Z 2022-11-23T02:13:37.8628305Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8628402Z Ran 1 test in 5.678s 2022-11-23T02:13:37.8628408Z 2022-11-23T02:13:37.8628490Z OK 2022-11-23T02:13:37.8628496Z 2022-11-23T02:13:37.8628607Z Generating XML reports... 2022-11-23T02:13:37.8629046Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021120.xml 2022-11-23T02:13:37.8629355Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8629724Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8629886Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8630264Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8630447Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8630453Z 2022-11-23T02:13:37.8630549Z Running tests... 2022-11-23T02:13:37.8630810Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8631123Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44441 2022-11-23T02:13:37.8631328Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44442 2022-11-23T02:13:37.8631576Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8631948Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8632111Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8632485Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8632662Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8632882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8633248Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8633409Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8633784Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8633958Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8634176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8634573Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8635026Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8635239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8635567Z STAGE:2022-11-23 02:11:33 44442:44442 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8635777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8636105Z STAGE:2022-11-23 02:11:33 44441:44441 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8636441Z STAGE:2022-11-23 02:11:33 44442:44442 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8636788Z STAGE:2022-11-23 02:11:33 44442:44442 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8637171Z STAGE:2022-11-23 02:11:33 44441:44441 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8637528Z STAGE:2022-11-23 02:11:33 44441:44441 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8637618Z ok (5.220s) 2022-11-23T02:13:37.8637624Z 2022-11-23T02:13:37.8637889Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8637986Z Ran 1 test in 5.220s 2022-11-23T02:13:37.8637992Z 2022-11-23T02:13:37.8638074Z OK 2022-11-23T02:13:37.8638079Z 2022-11-23T02:13:37.8638192Z Generating XML reports... 2022-11-23T02:13:37.8638623Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021130.xml 2022-11-23T02:13:37.8638932Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8639299Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8639468Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8639846Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8640019Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8640026Z 2022-11-23T02:13:37.8640124Z Running tests... 2022-11-23T02:13:37.8640386Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8640604Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.001s) 2022-11-23T02:13:37.8640610Z 2022-11-23T02:13:37.8640867Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8640963Z Ran 1 test in 0.002s 2022-11-23T02:13:37.8640969Z 2022-11-23T02:13:37.8641062Z OK (skipped=1) 2022-11-23T02:13:37.8641067Z 2022-11-23T02:13:37.8641180Z Generating XML reports... 2022-11-23T02:13:37.8641622Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021139.xml 2022-11-23T02:13:37.8641936Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8642306Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8642468Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8642842Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8643017Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8643023Z 2022-11-23T02:13:37.8643119Z Running tests... 2022-11-23T02:13:37.8643380Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8643615Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.001s) 2022-11-23T02:13:37.8643758Z 2022-11-23T02:13:37.8644017Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8644114Z Ran 1 test in 0.002s 2022-11-23T02:13:37.8644120Z 2022-11-23T02:13:37.8644215Z OK (skipped=1) 2022-11-23T02:13:37.8644221Z 2022-11-23T02:13:37.8644332Z Generating XML reports... 2022-11-23T02:13:37.8644770Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021143.xml 2022-11-23T02:13:37.8645079Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8645446Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8645610Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8645993Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8646223Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8646231Z 2022-11-23T02:13:37.8646329Z Running tests... 2022-11-23T02:13:37.8646596Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8646835Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s) 2022-11-23T02:13:37.8646841Z 2022-11-23T02:13:37.8647102Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8647202Z Ran 1 test in 0.002s 2022-11-23T02:13:37.8647208Z 2022-11-23T02:13:37.8647302Z OK (skipped=1) 2022-11-23T02:13:37.8647307Z 2022-11-23T02:13:37.8647419Z Generating XML reports... 2022-11-23T02:13:37.8647909Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021147.xml 2022-11-23T02:13:37.8648218Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8648599Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8648761Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8649140Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8649306Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8649322Z 2022-11-23T02:13:37.8649409Z Running tests... 2022-11-23T02:13:37.8649670Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8649978Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44852 2022-11-23T02:13:37.8650182Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44853 2022-11-23T02:13:37.8650434Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8650803Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8650966Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8651344Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8651517Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8651739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8652104Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8652262Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8652646Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8652884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8653104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8653501Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8653886Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8654100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8654312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8654639Z STAGE:2022-11-23 02:11:54 44852:44852 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8655018Z STAGE:2022-11-23 02:11:54 44853:44853 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8655362Z STAGE:2022-11-23 02:11:54 44853:44853 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8655707Z STAGE:2022-11-23 02:11:54 44853:44853 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8656031Z STAGE:2022-11-23 02:11:54 44852:44852 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8656376Z STAGE:2022-11-23 02:11:54 44852:44852 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8656466Z ok (4.864s) 2022-11-23T02:13:37.8656472Z 2022-11-23T02:13:37.8656734Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8656833Z Ran 1 test in 4.865s 2022-11-23T02:13:37.8656839Z 2022-11-23T02:13:37.8656920Z OK 2022-11-23T02:13:37.8656925Z 2022-11-23T02:13:37.8657036Z Generating XML reports... 2022-11-23T02:13:37.8657475Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021151.xml 2022-11-23T02:13:37.8657784Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8658153Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8658314Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8658691Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8658865Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8658871Z 2022-11-23T02:13:37.8658966Z Running tests... 2022-11-23T02:13:37.8659230Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8659526Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45065 2022-11-23T02:13:37.8659736Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45066 2022-11-23T02:13:37.8659987Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8660355Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8660516Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8660891Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8661066Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8661278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8661643Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8661866Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8662249Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8662424Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8662646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8663035Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8663424Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8663633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8663843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8663981Z ok (5.023s) 2022-11-23T02:13:37.8663988Z 2022-11-23T02:13:37.8664254Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8664351Z Ran 1 test in 5.023s 2022-11-23T02:13:37.8664357Z 2022-11-23T02:13:37.8664438Z OK 2022-11-23T02:13:37.8664443Z 2022-11-23T02:13:37.8664555Z Generating XML reports... 2022-11-23T02:13:37.8664993Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021200.xml 2022-11-23T02:13:37.8665299Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8665667Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8665829Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8666209Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8666391Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8666397Z 2022-11-23T02:13:37.8666493Z Running tests... 2022-11-23T02:13:37.8666747Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8667069Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45272 2022-11-23T02:13:37.8667275Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45273 2022-11-23T02:13:37.8667527Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8667895Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8668055Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8668442Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8668618Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8668843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8669206Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8669366Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8669743Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8669918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8670140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8670536Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8670980Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8671193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8671522Z STAGE:2022-11-23 02:12:12 45273:45273 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8671734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8672061Z STAGE:2022-11-23 02:12:12 45272:45272 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8672394Z STAGE:2022-11-23 02:12:12 45272:45272 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8672738Z STAGE:2022-11-23 02:12:12 45272:45272 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8673122Z STAGE:2022-11-23 02:12:12 45273:45273 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8673462Z STAGE:2022-11-23 02:12:12 45273:45273 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8673553Z ok (5.015s) 2022-11-23T02:13:37.8673559Z 2022-11-23T02:13:37.8673823Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8673920Z Ran 1 test in 5.015s 2022-11-23T02:13:37.8673926Z 2022-11-23T02:13:37.8674007Z OK 2022-11-23T02:13:37.8674012Z 2022-11-23T02:13:37.8674124Z Generating XML reports... 2022-11-23T02:13:37.8674562Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021209.xml 2022-11-23T02:13:37.8674868Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8675237Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8675407Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8675784Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8675959Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8675965Z 2022-11-23T02:13:37.8676060Z Running tests... 2022-11-23T02:13:37.8676323Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8676640Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45485 2022-11-23T02:13:37.8676844Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45486 2022-11-23T02:13:37.8677098Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8677468Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8677632Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8678008Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8678182Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8678400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8678792Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8679147Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8679308Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8679688Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8679936Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8680155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8680548Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8680764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8681089Z STAGE:2022-11-23 02:12:21 45486:45486 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8681299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8681626Z STAGE:2022-11-23 02:12:21 45485:45485 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:13:37.8682003Z STAGE:2022-11-23 02:12:21 45486:45486 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8682360Z STAGE:2022-11-23 02:12:21 45486:45486 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8682691Z STAGE:2022-11-23 02:12:21 45485:45485 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:13:37.8683037Z STAGE:2022-11-23 02:12:21 45485:45485 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:13:37.8683127Z ok (5.323s) 2022-11-23T02:13:37.8683134Z 2022-11-23T02:13:37.8683396Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8683493Z Ran 1 test in 5.323s 2022-11-23T02:13:37.8683499Z 2022-11-23T02:13:37.8683580Z OK 2022-11-23T02:13:37.8683586Z 2022-11-23T02:13:37.8683696Z Generating XML reports... 2022-11-23T02:13:37.8684136Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021219.xml 2022-11-23T02:13:37.8684454Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8684826Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8684978Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8685357Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8685530Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8685536Z 2022-11-23T02:13:37.8685632Z Running tests... 2022-11-23T02:13:37.8685893Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8686197Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45698 2022-11-23T02:13:37.8686401Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45699 2022-11-23T02:13:37.8686658Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8687026Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8687189Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8687568Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8687862Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8688088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8688463Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8688624Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8689085Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8689257Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8689478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8689870Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8690258Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8690469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8690679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8690770Z ok (4.945s) 2022-11-23T02:13:37.8690777Z 2022-11-23T02:13:37.8691089Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8691190Z Ran 1 test in 4.945s 2022-11-23T02:13:37.8691195Z 2022-11-23T02:13:37.8691276Z OK 2022-11-23T02:13:37.8691282Z 2022-11-23T02:13:37.8691395Z Generating XML reports... 2022-11-23T02:13:37.8691833Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021228.xml 2022-11-23T02:13:37.8692139Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8692506Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8692666Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8693047Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8693222Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8693236Z 2022-11-23T02:13:37.8693332Z Running tests... 2022-11-23T02:13:37.8693593Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8693900Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46091 2022-11-23T02:13:37.8694101Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46092 2022-11-23T02:13:37.8694354Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8694722Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8694885Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8695261Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8695439Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8695658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8696024Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8696184Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8696555Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8696729Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8696952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8697345Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8697797Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8698005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8698215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8698304Z ok (5.423s) 2022-11-23T02:13:37.8698310Z 2022-11-23T02:13:37.8698574Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8698671Z Ran 1 test in 5.423s 2022-11-23T02:13:37.8698677Z 2022-11-23T02:13:37.8698756Z OK 2022-11-23T02:13:37.8698762Z 2022-11-23T02:13:37.8698875Z Generating XML reports... 2022-11-23T02:13:37.8699314Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021237.xml 2022-11-23T02:13:37.8699627Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8700045Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8700211Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8700590Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8700764Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8700770Z 2022-11-23T02:13:37.8700866Z Running tests... 2022-11-23T02:13:37.8701129Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8701432Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46486 2022-11-23T02:13:37.8701635Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46487 2022-11-23T02:13:37.8701882Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8702253Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8702415Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8702790Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8702967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8703191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8703557Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8703720Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8704096Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8704272Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8704493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8704884Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8705269Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8705480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8705693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8705782Z ok (7.920s) 2022-11-23T02:13:37.8705788Z 2022-11-23T02:13:37.8706049Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8706203Z Ran 1 test in 7.920s 2022-11-23T02:13:37.8706214Z 2022-11-23T02:13:37.8706295Z OK 2022-11-23T02:13:37.8706301Z 2022-11-23T02:13:37.8706411Z Generating XML reports... 2022-11-23T02:13:37.8706848Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021247.xml 2022-11-23T02:13:37.8707154Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8707514Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8707679Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8708057Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8708235Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8708241Z 2022-11-23T02:13:37.8708339Z Running tests... 2022-11-23T02:13:37.8708651Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8708953Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46703 2022-11-23T02:13:37.8709156Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46704 2022-11-23T02:13:37.8709408Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8709779Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8709942Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8710319Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8710495Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8710728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8711094Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8711259Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8711635Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8711814Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8712037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8712433Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8712820Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8713039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8713248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8713472Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphy46w5hc 2022-11-23T02:13:37.8713718Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphy46w5hc/_remote_module_non_scriptable.py 2022-11-23T02:13:37.8713948Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnh8ekmc9 2022-11-23T02:13:37.8714197Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnh8ekmc9/_remote_module_non_scriptable.py 2022-11-23T02:13:37.8714287Z ok (5.068s) 2022-11-23T02:13:37.8714293Z 2022-11-23T02:13:37.8714554Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8714651Z Ran 1 test in 5.069s 2022-11-23T02:13:37.8714657Z 2022-11-23T02:13:37.8714793Z OK 2022-11-23T02:13:37.8714803Z 2022-11-23T02:13:37.8714916Z Generating XML reports... 2022-11-23T02:13:37.8715363Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021259.xml 2022-11-23T02:13:37.8715677Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8716050Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8716213Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8716591Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8716767Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8716772Z 2022-11-23T02:13:37.8716869Z Running tests... 2022-11-23T02:13:37.8717132Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8717476Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46976 2022-11-23T02:13:37.8717679Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46977 2022-11-23T02:13:37.8717930Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8718302Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8718466Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8718833Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8719007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8719239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8719630Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8719992Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8720153Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8720533Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8720706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8720929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8721319Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8721535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8721751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8721981Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1hnfqg6g 2022-11-23T02:13:37.8722229Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1hnfqg6g/_remote_module_non_scriptable.py 2022-11-23T02:13:37.8722461Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbz4h1nkh 2022-11-23T02:13:37.8722709Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbz4h1nkh/_remote_module_non_scriptable.py 2022-11-23T02:13:37.8722799Z ok (5.420s) 2022-11-23T02:13:37.8722805Z 2022-11-23T02:13:37.8723068Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8723165Z Ran 1 test in 5.420s 2022-11-23T02:13:37.8723171Z 2022-11-23T02:13:37.8723252Z OK 2022-11-23T02:13:37.8723258Z 2022-11-23T02:13:37.8723428Z Generating XML reports... 2022-11-23T02:13:37.8723869Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021308.xml 2022-11-23T02:13:37.8724171Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8724538Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8724699Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8725078Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8725252Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8725258Z 2022-11-23T02:13:37.8725354Z Running tests... 2022-11-23T02:13:37.8725615Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8725994Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47183 2022-11-23T02:13:37.8726204Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47184 2022-11-23T02:13:37.8726454Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:13:37.8726824Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8726985Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8727361Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8727537Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8727807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:13:37.8728182Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8728343Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8728724Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8728900Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8729122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:13:37.8729513Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8729900Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:13:37.8730114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:13:37.8730320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:13:37.8730552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvqi5yikv 2022-11-23T02:13:37.8730800Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvqi5yikv/_remote_module_non_scriptable.py 2022-11-23T02:13:37.8731034Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx5rf9ng0 2022-11-23T02:13:37.8731278Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx5rf9ng0/_remote_module_non_scriptable.py 2022-11-23T02:13:37.8732037Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:13:37.8732861Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:13:37.8732955Z ok (7.323s) 2022-11-23T02:13:37.8732961Z 2022-11-23T02:13:37.8733230Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8733319Z Ran 1 test in 7.324s 2022-11-23T02:13:37.8733334Z 2022-11-23T02:13:37.8733407Z OK 2022-11-23T02:13:37.8733417Z 2022-11-23T02:13:37.8733575Z Generating XML reports... 2022-11-23T02:13:37.8734022Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021317.xml 2022-11-23T02:13:37.8734332Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8734702Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8734866Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8735244Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8735420Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8735426Z 2022-11-23T02:13:37.8735522Z Running tests... 2022-11-23T02:13:37.8735786Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8736251Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.002s) 2022-11-23T02:13:37.8736258Z 2022-11-23T02:13:37.8736519Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8736616Z Ran 1 test in 0.002s 2022-11-23T02:13:37.8736622Z 2022-11-23T02:13:37.8736715Z OK (skipped=1) 2022-11-23T02:13:37.8736720Z 2022-11-23T02:13:37.8736832Z Generating XML reports... 2022-11-23T02:13:37.8737270Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021329.xml 2022-11-23T02:13:37.8737582Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:13:37.8737952Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:13:37.8738116Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:13:37.8738498Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:13:37.8738674Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:13:37.8738682Z 2022-11-23T02:13:37.8738777Z Running tests... 2022-11-23T02:13:37.8739031Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8739498Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.001s) 2022-11-23T02:13:37.8739517Z 2022-11-23T02:13:37.8739768Z ---------------------------------------------------------------------- 2022-11-23T02:13:37.8739864Z Ran 1 test in 0.002s 2022-11-23T02:13:37.8739870Z 2022-11-23T02:13:37.8739968Z OK (skipped=1) 2022-11-23T02:13:37.8739974Z 2022-11-23T02:13:37.8740085Z Generating XML reports... 2022-11-23T02:13:37.8740526Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021333.xml 2022-11-23T02:13:37.8740590Z 2022-11-23T02:13:37.8741078Z ##[endgroup] 2022-11-23T02:13:37.8741548Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_distributed_spawn_n0h3_4pw) 2022-11-23T02:13:37.8741555Z 2022-11-23T02:13:37.8741750Z Running distributed tests for the gloo backend with file init_method in shard 1 of 2 2022-11-23T02:13:37.8742273Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:13:37.565048] 2022-11-23T02:49:35.2747751Z 2022-11-23T02:49:35.2748594Z Expand the folded group to see the log file of distributed/test_distributed_spawn 2022-11-23T02:49:35.2750735Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_distributed_spawn_xw60i0rv) 2022-11-23T02:49:35.2760181Z 2022-11-23T02:49:35.2857661Z , <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_cat_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_stack_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_zero_output_features>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]> 2022-11-23T02:49:35.2921920Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2922788Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2923529Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2924264Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2925022Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2925832Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2926682Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2927506Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2928467Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2929554Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2930541Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2931413Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2932350Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2933348Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2934294Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2935232Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2936058Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2936826Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2937604Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2938442Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2939308Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2940213Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2940958Z test_all_gather (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2941689Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2942495Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2943286Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2944080Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2944871Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2945615Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2946352Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2947089Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2947884Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2948613Z test_all_gather_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2949352Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2950139Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2950903Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2951654Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2952425Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2953219Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2954180Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2954940Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2955774Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2956559Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2957378Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2958189Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2958911Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2959970Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2961016Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2962006Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2963141Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2964120Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2965005Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2965851Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2966728Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2967593Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2968610Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2969452Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2970407Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2971335Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2972206Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2973128Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2974036Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2974930Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2975798Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2976655Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2977541Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2978375Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2979213Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2980018Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2980858Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2981702Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2982526Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2983306Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2984083Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2984819Z test_all_to_all (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2985499Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2986409Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2987345Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2988274Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2989244Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2990086Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2990816Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2991831Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2992672Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2993571Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2994637Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2995547Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2996506Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2997464Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2998529Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.2999589Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3000617Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3001757Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3002878Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3003849Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3004842Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3005695Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3006616Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3007359Z test_average_parameters (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3008115Z test_backend_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3008742Z test_backend_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3009391Z test_barrier (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3010017Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3010667Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3011463Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3011910Z test_barrier_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3012369Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3012832Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3013314Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3013770Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3014237Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3014703Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3015210Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3015703Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3016194Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3016659Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3017138Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3017650Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3018132Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3018618Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3019059Z test_broadcast (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3019494Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3019921Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3020516Z test_broadcast_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3020994Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3021446Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3021978Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3022571Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3023088Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3023554Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3024045Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3024539Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3025061Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3025685Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3026209Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3026674Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3027206Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3027677Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3028112Z test_ddp_device (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3028580Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3029053Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3029543Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3030034Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3030560Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3031063Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3031546Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3032099Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3032694Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3033359Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3034079Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3034804Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3035533Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3036313Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3037015Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3037734Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3038447Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3039090Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3039747Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3040277Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3040712Z test_ddp_inference (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3041242Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3041711Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3042186Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3042672Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3043185Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3043725Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3044268Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3044812Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3045283Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3045755Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3046267Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3046777Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3047250Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3047851Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3048367Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3048883Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3049378Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3049847Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3050345Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3050826Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3051302Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3051796Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3052367Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3052879Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3053360Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3053790Z test_destroy_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3054263Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3054751Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3055249Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3055689Z test_gather (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3056135Z test_gather_checks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3056567Z test_gather_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3056992Z test_gather_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3057443Z test_gather_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3057880Z test_gather_object (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3058347Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3058808Z test_get_backend (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3059225Z test_get_future (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3059627Z test_get_rank (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3060069Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3060643Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3061113Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3061540Z test_irecv (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3061957Z test_isend (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3062380Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3062856Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3063336Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3063869Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3064383Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3064931Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3065692Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3066406Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3066903Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3067395Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3067885Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3068359Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3068842Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3069305Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3069764Z test_new_subgroups (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3070230Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3070768Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3071341Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3071888Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3072410Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3072940Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3073472Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3073970Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3074458Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3074954Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3075466Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3075990Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3076535Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3077127Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3077687Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3078179Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3078638Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3079107Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3079561Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3080017Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3080459Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3080913Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3081442Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3081874Z test_reduce_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3082298Z test_reduce_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3082717Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3083159Z test_reduce_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3083619Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3084084Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3084514Z test_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3084943Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3085378Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3085829Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3086309Z test_scatter (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3086742Z test_scatter_checks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3087177Z test_scatter_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3087616Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3088236Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3088703Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3089142Z test_scatter_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3089590Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3090022Z test_send_recv (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3090454Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3090927Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3091438Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3091942Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3092399Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3092868Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3093363Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3093838Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3094282Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3094756Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3095261Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3095744Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3096208Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3096682Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3097136Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3097580Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3098061Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3098577Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3099084Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3099936Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3100700Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3101215Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3101914Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3102566Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3102828Z 2022-11-23T02:49:35.3102944Z Running tests... 2022-11-23T02:49:35.3103451Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3104171Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47598 2022-11-23T02:49:35.3104893Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47599 2022-11-23T02:49:35.3105484Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3106248Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3106779Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3107547Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3108103Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3108623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3109381Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3109896Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3110580Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3111127Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3111653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3112437Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3113259Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3113858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3114401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3114997Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:49:35.3115977Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:49:35.3116743Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:49:35.3117733Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:49:35.3118487Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:49:35.3119474Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:49:35.3120235Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:49:35.3121210Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:49:35.3121970Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:49:35.3123024Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:49:35.3123771Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:49:35.3124736Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:49:35.3125486Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:49:35.3126459Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:49:35.3127204Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:49:35.3128537Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:49:35.3129099Z ok (5.620s) 2022-11-23T02:49:35.3129270Z 2022-11-23T02:49:35.3129611Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3129987Z Ran 1 test in 5.621s 2022-11-23T02:49:35.3130168Z 2022-11-23T02:49:35.3130255Z OK 2022-11-23T02:49:35.3130407Z 2022-11-23T02:49:35.3130538Z Generating XML reports... 2022-11-23T02:49:35.3131273Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021341.xml 2022-11-23T02:49:35.3132040Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3132788Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3133331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3134024Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3134552Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3134812Z 2022-11-23T02:49:35.3134931Z Running tests... 2022-11-23T02:49:35.3135419Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3135992Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.004s) 2022-11-23T02:49:35.3136341Z 2022-11-23T02:49:35.3136659Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3137029Z Ran 1 test in 0.004s 2022-11-23T02:49:35.3137212Z 2022-11-23T02:49:35.3137325Z OK (skipped=1) 2022-11-23T02:49:35.3137498Z 2022-11-23T02:49:35.3137625Z Generating XML reports... 2022-11-23T02:49:35.3138345Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021351.xml 2022-11-23T02:49:35.3139117Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3139865Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3140385Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3141079Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3141619Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3141876Z 2022-11-23T02:49:35.3141993Z Running tests... 2022-11-23T02:49:35.3142468Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3143087Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47875 2022-11-23T02:49:35.3143747Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47876 2022-11-23T02:49:35.3144235Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3144893Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3145332Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3145909Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3146349Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3146778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3147553Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3147999Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3148583Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3149033Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3149462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3150108Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3150776Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3151277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3151738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3152073Z ok (4.921s) 2022-11-23T02:49:35.3152209Z 2022-11-23T02:49:35.3152476Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3152789Z Ran 1 test in 4.921s 2022-11-23T02:49:35.3152937Z 2022-11-23T02:49:35.3153020Z OK 2022-11-23T02:49:35.3153131Z 2022-11-23T02:49:35.3153245Z Generating XML reports... 2022-11-23T02:49:35.3153840Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021355.xml 2022-11-23T02:49:35.3154487Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3155107Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3155543Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3156118Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3156572Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3156785Z 2022-11-23T02:49:35.3156873Z Running tests... 2022-11-23T02:49:35.3157283Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3158432Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.588s) 2022-11-23T02:49:35.3159055Z 2022-11-23T02:49:35.3159322Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3159634Z Ran 1 test in 0.589s 2022-11-23T02:49:35.3159783Z 2022-11-23T02:49:35.3159879Z OK (skipped=1) 2022-11-23T02:49:35.3160022Z 2022-11-23T02:49:35.3160218Z Generating XML reports... 2022-11-23T02:49:35.3160819Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021404.xml 2022-11-23T02:49:35.3161464Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3162073Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3162508Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3163083Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3163535Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3163749Z 2022-11-23T02:49:35.3163849Z Running tests... 2022-11-23T02:49:35.3164252Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3164840Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48148 2022-11-23T02:49:35.3165400Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48149 2022-11-23T02:49:35.3165880Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3166534Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3166971Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3167545Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3168066Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3168493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3169259Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3170041Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3170565Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3171251Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3171799Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3172320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3173102Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3173703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3174278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6gzoiwl1 2022-11-23T02:49:35.3174878Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6gzoiwl1/_remote_module_non_scriptable.py 2022-11-23T02:49:35.3175463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3176035Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3f5t91gb 2022-11-23T02:49:35.3176646Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3f5t91gb/_remote_module_non_scriptable.py 2022-11-23T02:49:35.3177233Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3177786Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3178193Z ok (5.216s) 2022-11-23T02:49:35.3178353Z 2022-11-23T02:49:35.3178666Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3179131Z Ran 1 test in 5.216s 2022-11-23T02:49:35.3179320Z 2022-11-23T02:49:35.3179418Z OK 2022-11-23T02:49:35.3179569Z 2022-11-23T02:49:35.3179703Z Generating XML reports... 2022-11-23T02:49:35.3180425Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021409.xml 2022-11-23T02:49:35.3181192Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3181936Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3182443Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3183137Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3183665Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3183882Z 2022-11-23T02:49:35.3184044Z Running tests... 2022-11-23T02:49:35.3184459Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3185012Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48421 2022-11-23T02:49:35.3185577Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48422 2022-11-23T02:49:35.3186056Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3186702Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3187141Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3187713Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3188170Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3188606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3189227Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3189661Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3190222Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3190674Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3191107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3191757Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3192436Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3192935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3193410Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3q0uo0fe 2022-11-23T02:49:35.3193920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3q0uo0fe/_remote_module_non_scriptable.py 2022-11-23T02:49:35.3194395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3194873Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9rux4gve 2022-11-23T02:49:35.3195381Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9rux4gve/_remote_module_non_scriptable.py 2022-11-23T02:49:35.3195875Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3196342Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3196736Z ok (5.520s) 2022-11-23T02:49:35.3196873Z 2022-11-23T02:49:35.3197146Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3197448Z Ran 1 test in 5.520s 2022-11-23T02:49:35.3197600Z 2022-11-23T02:49:35.3197683Z OK 2022-11-23T02:49:35.3197808Z 2022-11-23T02:49:35.3197922Z Generating XML reports... 2022-11-23T02:49:35.3198518Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021419.xml 2022-11-23T02:49:35.3199163Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3199785Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3200221Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3200842Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3201301Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3201516Z 2022-11-23T02:49:35.3201615Z Running tests... 2022-11-23T02:49:35.3202024Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3202572Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48694 2022-11-23T02:49:35.3203127Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48695 2022-11-23T02:49:35.3203619Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3204252Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3204689Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3205271Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3205726Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3206159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3206803Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3207462Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3208052Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3208629Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3209174Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3209702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3210478Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3211075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3211620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3212196Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_dqpwbg1 2022-11-23T02:49:35.3212798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_dqpwbg1/_remote_module_non_scriptable.py 2022-11-23T02:49:35.3213409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpblv5bory 2022-11-23T02:49:35.3214019Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpblv5bory/_remote_module_non_scriptable.py 2022-11-23T02:49:35.3214782Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3215337Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3215884Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3216439Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3216978Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3217529Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3218083Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3218627Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3219180Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3219778Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3220327Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3220882Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3221274Z ok (8.125s) 2022-11-23T02:49:35.3221442Z 2022-11-23T02:49:35.3221767Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3222138Z Ran 1 test in 8.125s 2022-11-23T02:49:35.3222318Z 2022-11-23T02:49:35.3222416Z OK 2022-11-23T02:49:35.3222564Z 2022-11-23T02:49:35.3222700Z Generating XML reports... 2022-11-23T02:49:35.3223424Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021428.xml 2022-11-23T02:49:35.3224074Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3224701Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3225139Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3225714Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3226169Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3226384Z 2022-11-23T02:49:35.3226483Z Running tests... 2022-11-23T02:49:35.3226888Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3227439Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48911 2022-11-23T02:49:35.3228011Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48912 2022-11-23T02:49:35.3228504Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3229156Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3229595Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3230169Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3230624Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3231061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3231671Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3232105Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3232683Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3233198Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3233625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3234277Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3234956Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3235449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3235889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3236362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8i16x85e 2022-11-23T02:49:35.3236917Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8i16x85e/_remote_module_non_scriptable.py 2022-11-23T02:49:35.3237433Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl79sy9a6 2022-11-23T02:49:35.3237941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl79sy9a6/_remote_module_non_scriptable.py 2022-11-23T02:49:35.3238461Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3238989Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3239478Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3239924Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3240382Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3240840Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3241182Z ok (5.832s) 2022-11-23T02:49:35.3241317Z 2022-11-23T02:49:35.3241594Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3241908Z Ran 1 test in 5.833s 2022-11-23T02:49:35.3242059Z 2022-11-23T02:49:35.3242131Z OK 2022-11-23T02:49:35.3242253Z 2022-11-23T02:49:35.3242370Z Generating XML reports... 2022-11-23T02:49:35.3242969Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021441.xml 2022-11-23T02:49:35.3243611Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3244226Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3244661Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3245240Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3245694Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3245899Z 2022-11-23T02:49:35.3245998Z Running tests... 2022-11-23T02:49:35.3246402Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3246972Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49126 2022-11-23T02:49:35.3247559Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49127 2022-11-23T02:49:35.3248278Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3249010Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3249539Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3250313Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3250860Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3251379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3252121Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3252641Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3253333Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3253877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3254394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3255234Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3256073Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3256669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3257208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3257779Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdqbnpgsw 2022-11-23T02:49:35.3258402Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdqbnpgsw/_remote_module_non_scriptable.py 2022-11-23T02:49:35.3259012Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaz7hu4x0 2022-11-23T02:49:35.3259619Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaz7hu4x0/_remote_module_non_scriptable.py 2022-11-23T02:49:35.3260208Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3260758Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3261312Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3261863Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3262406Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3262955Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3263501Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3263983Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3264316Z ok (5.719s) 2022-11-23T02:49:35.3264460Z 2022-11-23T02:49:35.3264735Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3265050Z Ran 1 test in 5.719s 2022-11-23T02:49:35.3265200Z 2022-11-23T02:49:35.3265283Z OK 2022-11-23T02:49:35.3265407Z 2022-11-23T02:49:35.3265521Z Generating XML reports... 2022-11-23T02:49:35.3266109Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021451.xml 2022-11-23T02:49:35.3266752Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3267371Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3267804Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3268377Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3268834Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3269127Z 2022-11-23T02:49:35.3269226Z Running tests... 2022-11-23T02:49:35.3269622Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3270214Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49341 2022-11-23T02:49:35.3270814Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49342 2022-11-23T02:49:35.3271305Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3271954Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3272390Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3273014Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3273473Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3273889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3274512Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3274945Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3275514Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3275964Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3276392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3277042Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3277722Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3278207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3278663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3279136Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp42n2jjen 2022-11-23T02:49:35.3279642Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp42n2jjen/_remote_module_non_scriptable.py 2022-11-23T02:49:35.3280146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpem_ytv72 2022-11-23T02:49:35.3280651Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpem_ytv72/_remote_module_non_scriptable.py 2022-11-23T02:49:35.3281137Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3281585Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3281921Z ok (5.828s) 2022-11-23T02:49:35.3282059Z 2022-11-23T02:49:35.3282329Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3282643Z Ran 1 test in 5.829s 2022-11-23T02:49:35.3282792Z 2022-11-23T02:49:35.3282875Z OK 2022-11-23T02:49:35.3282998Z 2022-11-23T02:49:35.3283112Z Generating XML reports... 2022-11-23T02:49:35.3283706Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021501.xml 2022-11-23T02:49:35.3284336Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3284951Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3285389Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3286037Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3286494Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3286709Z 2022-11-23T02:49:35.3286807Z Running tests... 2022-11-23T02:49:35.3287211Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3287935Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49556 2022-11-23T02:49:35.3288538Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49557 2022-11-23T02:49:35.3289106Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3289976Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3290511Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3291209Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3291754Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3292274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3293013Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3293532Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3294215Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3294754Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3295281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3296058Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3296865Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3297457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3297992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3298554Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptubgm743 2022-11-23T02:49:35.3299165Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptubgm743/_remote_module_non_scriptable.py 2022-11-23T02:49:35.3299769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp318gevbq 2022-11-23T02:49:35.3300384Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp318gevbq/_remote_module_non_scriptable.py 2022-11-23T02:49:35.3300975Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3301519Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3302055Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3302593Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3302995Z ok (7.826s) 2022-11-23T02:49:35.3303161Z 2022-11-23T02:49:35.3303486Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3303850Z Ran 1 test in 7.827s 2022-11-23T02:49:35.3304003Z 2022-11-23T02:49:35.3304089Z OK 2022-11-23T02:49:35.3304212Z 2022-11-23T02:49:35.3304327Z Generating XML reports... 2022-11-23T02:49:35.3304916Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021511.xml 2022-11-23T02:49:35.3305617Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3306235Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3306671Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3307244Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3307698Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3307912Z 2022-11-23T02:49:35.3308011Z Running tests... 2022-11-23T02:49:35.3308405Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3309023Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49773 2022-11-23T02:49:35.3309600Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49774 2022-11-23T02:49:35.3310091Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3310741Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3311173Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3311750Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3312201Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3312622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3313249Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3313686Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3314254Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3314700Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3315132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3315773Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3316437Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3316935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3317392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3317876Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjxaypbk4 2022-11-23T02:49:35.3318387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjxaypbk4/_remote_module_non_scriptable.py 2022-11-23T02:49:35.3318894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8pyea1bo 2022-11-23T02:49:35.3319404Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8pyea1bo/_remote_module_non_scriptable.py 2022-11-23T02:49:35.3319891Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3320339Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3320793Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3321251Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3321649Z ok (7.792s) 2022-11-23T02:49:35.3321785Z 2022-11-23T02:49:35.3322056Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3322371Z Ran 1 test in 7.792s 2022-11-23T02:49:35.3322522Z 2022-11-23T02:49:35.3322594Z OK 2022-11-23T02:49:35.3322718Z 2022-11-23T02:49:35.3322833Z Generating XML reports... 2022-11-23T02:49:35.3323432Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021523.xml 2022-11-23T02:49:35.3324076Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3324691Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3325127Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3325703Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3326215Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3326421Z 2022-11-23T02:49:35.3326520Z Running tests... 2022-11-23T02:49:35.3326928Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3327514Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49990 2022-11-23T02:49:35.3328271Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49991 2022-11-23T02:49:35.3328836Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3329685Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3330253Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3330994Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3331594Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3332156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3332991Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3333862Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3334650Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3335727Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3336277Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3336792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3337572Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3338173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3338721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3339287Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjqqqapgp 2022-11-23T02:49:35.3339906Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjqqqapgp/_remote_module_non_scriptable.py 2022-11-23T02:49:35.3340518Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmparm_wggi 2022-11-23T02:49:35.3341118Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmparm_wggi/_remote_module_non_scriptable.py 2022-11-23T02:49:35.3341864Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3342497Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3343081Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3343632Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3344178Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3344729Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3345128Z ok (5.676s) 2022-11-23T02:49:35.3345278Z 2022-11-23T02:49:35.3345609Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3345979Z Ran 1 test in 5.676s 2022-11-23T02:49:35.3346165Z 2022-11-23T02:49:35.3346267Z OK 2022-11-23T02:49:35.3346481Z 2022-11-23T02:49:35.3346619Z Generating XML reports... 2022-11-23T02:49:35.3347347Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021535.xml 2022-11-23T02:49:35.3348110Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3348854Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3349367Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3350071Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3350615Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3350874Z 2022-11-23T02:49:35.3350994Z Running tests... 2022-11-23T02:49:35.3351476Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3352908Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.648s) 2022-11-23T02:49:35.3353673Z 2022-11-23T02:49:35.3353966Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3354281Z Ran 1 test in 0.648s 2022-11-23T02:49:35.3354432Z 2022-11-23T02:49:35.3354516Z OK (skipped=1) 2022-11-23T02:49:35.3354658Z 2022-11-23T02:49:35.3354777Z Generating XML reports... 2022-11-23T02:49:35.3355370Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021545.xml 2022-11-23T02:49:35.3356014Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3356633Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3357069Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3357646Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3358101Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3358305Z 2022-11-23T02:49:35.3358404Z Running tests... 2022-11-23T02:49:35.3358809Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3359356Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50271 2022-11-23T02:49:35.3359916Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50272 2022-11-23T02:49:35.3360470Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3361126Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3361559Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3362122Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3362579Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3363010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3363653Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3364309Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3364798Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3365377Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3365825Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3366246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3366895Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3367395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3367921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3368258Z ok (5.164s) 2022-11-23T02:49:35.3368395Z 2022-11-23T02:49:35.3368667Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3368990Z Ran 1 test in 5.165s 2022-11-23T02:49:35.3369129Z 2022-11-23T02:49:35.3369216Z OK 2022-11-23T02:49:35.3369337Z 2022-11-23T02:49:35.3369453Z Generating XML reports... 2022-11-23T02:49:35.3370049Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021550.xml 2022-11-23T02:49:35.3370686Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3371303Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3371733Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3372309Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3372755Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3372967Z 2022-11-23T02:49:35.3373074Z Running tests... 2022-11-23T02:49:35.3373479Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3374658Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.637s) 2022-11-23T02:49:35.3375278Z 2022-11-23T02:49:35.3375543Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3375858Z Ran 1 test in 0.637s 2022-11-23T02:49:35.3376009Z 2022-11-23T02:49:35.3376105Z OK (skipped=1) 2022-11-23T02:49:35.3376248Z 2022-11-23T02:49:35.3376363Z Generating XML reports... 2022-11-23T02:49:35.3376956Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021559.xml 2022-11-23T02:49:35.3377676Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3378292Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3378731Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3379308Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3379765Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3379977Z 2022-11-23T02:49:35.3380077Z Running tests... 2022-11-23T02:49:35.3380469Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3381003Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50544 2022-11-23T02:49:35.3381606Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50545 2022-11-23T02:49:35.3382100Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3382752Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3383188Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3383761Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3384212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3384631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3385277Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3385935Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3386366Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3386940Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3387390Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3387819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3388464Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3388957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3389413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3389749Z ok (5.647s) 2022-11-23T02:49:35.3389890Z 2022-11-23T02:49:35.3390156Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3390470Z Ran 1 test in 5.648s 2022-11-23T02:49:35.3390618Z 2022-11-23T02:49:35.3390700Z OK 2022-11-23T02:49:35.3390824Z 2022-11-23T02:49:35.3390943Z Generating XML reports... 2022-11-23T02:49:35.3391528Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021604.xml 2022-11-23T02:49:35.3392166Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3392784Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3393215Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3393791Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3394311Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3394528Z 2022-11-23T02:49:35.3394628Z Running tests... 2022-11-23T02:49:35.3395026Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3395483Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s) 2022-11-23T02:49:35.3395752Z 2022-11-23T02:49:35.3396018Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3396331Z Ran 1 test in 0.002s 2022-11-23T02:49:35.3396482Z 2022-11-23T02:49:35.3396578Z OK (skipped=1) 2022-11-23T02:49:35.3396720Z 2022-11-23T02:49:35.3396834Z Generating XML reports... 2022-11-23T02:49:35.3397430Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021613.xml 2022-11-23T02:49:35.3398062Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3398769Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3399209Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3399791Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3400241Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3400457Z 2022-11-23T02:49:35.3400558Z Running tests... 2022-11-23T02:49:35.3400963Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3401359Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3401864Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50817 2022-11-23T02:49:35.3402382Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50818 2022-11-23T02:49:35.3402881Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3403531Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3403964Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3404537Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3404990Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3405413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3406033Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3406465Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3407042Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3407501Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3408097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3408753Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3409420Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3409920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3410397Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb1364_ni 2022-11-23T02:49:35.3410902Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb1364_ni/_remote_module_non_scriptable.py 2022-11-23T02:49:35.3411467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3411946Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6zvgkh9n 2022-11-23T02:49:35.3412453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6zvgkh9n/_remote_module_non_scriptable.py 2022-11-23T02:49:35.3412822Z ok (5.156s) 2022-11-23T02:49:35.3412950Z 2022-11-23T02:49:35.3413223Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3413536Z Ran 1 test in 5.156s 2022-11-23T02:49:35.3413686Z 2022-11-23T02:49:35.3413771Z OK 2022-11-23T02:49:35.3413892Z 2022-11-23T02:49:35.3414008Z Generating XML reports... 2022-11-23T02:49:35.3414602Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021618.xml 2022-11-23T02:49:35.3415238Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3415909Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3416352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3416931Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3417382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3417595Z 2022-11-23T02:49:35.3417697Z Running tests... 2022-11-23T02:49:35.3418103Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3418536Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3419062Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51090 2022-11-23T02:49:35.3419587Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51091 2022-11-23T02:49:35.3420083Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3420727Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3421161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3421733Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3422184Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3422612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3423242Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3423906Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3424340Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3424914Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3425368Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3425800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3426444Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3426942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3427416Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp125urae 2022-11-23T02:49:35.3427927Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp125urae/_remote_module_non_scriptable.py 2022-11-23T02:49:35.3428476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3428955Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpifx0nzdb 2022-11-23T02:49:35.3429467Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpifx0nzdb/_remote_module_non_scriptable.py 2022-11-23T02:49:35.3429954Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3430416Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3430740Z ok (5.167s) 2022-11-23T02:49:35.3430876Z 2022-11-23T02:49:35.3431151Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3431463Z Ran 1 test in 5.167s 2022-11-23T02:49:35.3431616Z 2022-11-23T02:49:35.3431699Z OK 2022-11-23T02:49:35.3431823Z 2022-11-23T02:49:35.3431936Z Generating XML reports... 2022-11-23T02:49:35.3432728Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021627.xml 2022-11-23T02:49:35.3433501Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3434226Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3434743Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3435437Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3435976Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3436235Z 2022-11-23T02:49:35.3436351Z Running tests... 2022-11-23T02:49:35.3436834Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3437366Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3438033Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51363 2022-11-23T02:49:35.3438684Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51364 2022-11-23T02:49:35.3439273Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3440052Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3440574Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3441269Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3441822Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3442347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3443113Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3443912Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3444406Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3444984Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3445433Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3445864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3446513Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3447061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3447541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ud5qjwr 2022-11-23T02:49:35.3448176Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ud5qjwr/_remote_module_non_scriptable.py 2022-11-23T02:49:35.3448658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3449133Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdhdie6o1 2022-11-23T02:49:35.3449645Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdhdie6o1/_remote_module_non_scriptable.py 2022-11-23T02:49:35.3450136Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3450595Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.3450920Z ok (5.164s) 2022-11-23T02:49:35.3451132Z 2022-11-23T02:49:35.3451423Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3451739Z Ran 1 test in 5.165s 2022-11-23T02:49:35.3451888Z 2022-11-23T02:49:35.3451973Z OK 2022-11-23T02:49:35.3452097Z 2022-11-23T02:49:35.3452214Z Generating XML reports... 2022-11-23T02:49:35.3452806Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021636.xml 2022-11-23T02:49:35.3453430Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3454047Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3454483Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3455052Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3455512Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3455726Z 2022-11-23T02:49:35.3455824Z Running tests... 2022-11-23T02:49:35.3456232Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3456652Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:49:35.3457171Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51636 2022-11-23T02:49:35.3457685Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51637 2022-11-23T02:49:35.3458172Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3458817Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3459245Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3459824Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3460279Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3460696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3461316Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3461743Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3462321Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3462767Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3463196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3463914Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3464592Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3465080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3465532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3466008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0wrvgnx8 2022-11-23T02:49:35.3466513Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0wrvgnx8/_remote_module_non_scriptable.py 2022-11-23T02:49:35.3467017Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6or76hsc 2022-11-23T02:49:35.3467526Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6or76hsc/_remote_module_non_scriptable.py 2022-11-23T02:49:35.3467943Z ok (5.180s) 2022-11-23T02:49:35.3468082Z 2022-11-23T02:49:35.3468344Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3468662Z Ran 1 test in 5.181s 2022-11-23T02:49:35.3468810Z 2022-11-23T02:49:35.3468895Z OK 2022-11-23T02:49:35.3469016Z 2022-11-23T02:49:35.3469130Z Generating XML reports... 2022-11-23T02:49:35.3469721Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021646.xml 2022-11-23T02:49:35.3470361Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3470974Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3471396Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3471969Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3472430Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3472649Z 2022-11-23T02:49:35.3472746Z Running tests... 2022-11-23T02:49:35.3473150Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3473652Z test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51909 2022-11-23T02:49:35.3474162Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51910 2022-11-23T02:49:35.3474634Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3475279Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3475713Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3476290Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3476744Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3477171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3477816Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3478468Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3478892Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3479467Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3479917Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3480350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3481065Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3481562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3482139Z STAGE:2022-11-23 02:16:58 51910:51910 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3482594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3483158Z STAGE:2022-11-23 02:16:58 51909:51909 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3483739Z STAGE:2022-11-23 02:16:58 51910:51910 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3484320Z STAGE:2022-11-23 02:16:58 51909:51909 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3484964Z STAGE:2022-11-23 02:16:58 51910:51910 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3485582Z STAGE:2022-11-23 02:16:58 51909:51909 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3486155Z STAGE:2022-11-23 02:16:58 51909:51909 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3486721Z STAGE:2022-11-23 02:16:58 51910:51910 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3487283Z STAGE:2022-11-23 02:16:58 51909:51909 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3488098Z STAGE:2022-11-23 02:16:58 51909:51909 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3488701Z STAGE:2022-11-23 02:16:58 51910:51910 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3489292Z STAGE:2022-11-23 02:16:58 51910:51910 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3489641Z ok (5.469s) 2022-11-23T02:49:35.3489783Z 2022-11-23T02:49:35.3490050Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3490358Z Ran 1 test in 5.470s 2022-11-23T02:49:35.3490509Z 2022-11-23T02:49:35.3490581Z OK 2022-11-23T02:49:35.3490708Z 2022-11-23T02:49:35.3490822Z Generating XML reports... 2022-11-23T02:49:35.3491416Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021655.xml 2022-11-23T02:49:35.3492052Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3492665Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3492832Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3493211Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3493394Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3493400Z 2022-11-23T02:49:35.3493499Z Running tests... 2022-11-23T02:49:35.3493767Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3494083Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52122 2022-11-23T02:49:35.3494289Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52123 2022-11-23T02:49:35.3494539Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3494910Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3495064Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3495449Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3495744Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3495968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3496341Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3496505Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3496884Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3497061Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3497282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3497728Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3498131Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3498344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3498675Z STAGE:2022-11-23 02:17:07 52123:52123 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3498890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3499220Z STAGE:2022-11-23 02:17:07 52122:52122 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3499556Z STAGE:2022-11-23 02:17:07 52122:52122 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3499906Z STAGE:2022-11-23 02:17:07 52122:52122 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3500240Z STAGE:2022-11-23 02:17:07 52123:52123 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3500590Z STAGE:2022-11-23 02:17:07 52123:52123 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3500914Z STAGE:2022-11-23 02:17:07 52122:52122 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3501238Z STAGE:2022-11-23 02:17:07 52123:52123 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3501786Z STAGE:2022-11-23 02:17:07 52123:52123 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:17:07 52122:52122 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3501793Z 2022-11-23T02:49:35.3502141Z STAGE:2022-11-23 02:17:07 52123:52123 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3502488Z STAGE:2022-11-23 02:17:07 52122:52122 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3502823Z STAGE:2022-11-23 02:17:07 52123:52123 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3503148Z STAGE:2022-11-23 02:17:07 52122:52122 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3503471Z STAGE:2022-11-23 02:17:07 52122:52122 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3503821Z STAGE:2022-11-23 02:17:07 52122:52122 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3504153Z STAGE:2022-11-23 02:17:07 52123:52123 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3504499Z STAGE:2022-11-23 02:17:07 52123:52123 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3505255Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:49:35.3505413Z warnings.warn( 2022-11-23T02:49:35.3506160Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:49:35.3506259Z warnings.warn( 2022-11-23T02:49:35.3506350Z ok (5.045s) 2022-11-23T02:49:35.3506356Z 2022-11-23T02:49:35.3506622Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3506722Z Ran 1 test in 5.045s 2022-11-23T02:49:35.3506729Z 2022-11-23T02:49:35.3506811Z OK 2022-11-23T02:49:35.3506816Z 2022-11-23T02:49:35.3506929Z Generating XML reports... 2022-11-23T02:49:35.3507370Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021704.xml 2022-11-23T02:49:35.3507726Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3508109Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3508277Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3508659Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3508835Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3508841Z 2022-11-23T02:49:35.3508939Z Running tests... 2022-11-23T02:49:35.3509205Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3509521Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52335 2022-11-23T02:49:35.3509727Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52336 2022-11-23T02:49:35.3509988Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3510359Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3510512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3510890Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3511067Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3511292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3511658Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3511822Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3512206Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3512381Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3512604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3512999Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3513389Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3513604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3513816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3514039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:49:35.3514335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:49:35.3514729Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.3515120Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.3515449Z STAGE:2022-11-23 02:17:17 52335:52335 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3515776Z STAGE:2022-11-23 02:17:17 52336:52336 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3516110Z STAGE:2022-11-23 02:17:17 52335:52335 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3516461Z STAGE:2022-11-23 02:17:17 52335:52335 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3516835Z STAGE:2022-11-23 02:17:17 52336:52336 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3517191Z STAGE:2022-11-23 02:17:17 52336:52336 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3517518Z STAGE:2022-11-23 02:17:17 52335:52335 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3517832Z STAGE:2022-11-23 02:17:17 52336:52336 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3518165Z STAGE:2022-11-23 02:17:17 52335:52335 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3518514Z STAGE:2022-11-23 02:17:17 52335:52335 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3518843Z STAGE:2022-11-23 02:17:17 52336:52336 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3519189Z STAGE:2022-11-23 02:17:17 52336:52336 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3519519Z STAGE:2022-11-23 02:17:17 52335:52335 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3519847Z STAGE:2022-11-23 02:17:17 52336:52336 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3520178Z STAGE:2022-11-23 02:17:17 52335:52335 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3520525Z STAGE:2022-11-23 02:17:17 52335:52335 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3520855Z STAGE:2022-11-23 02:17:17 52336:52336 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3521201Z STAGE:2022-11-23 02:17:17 52336:52336 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3521954Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:49:35.3522058Z warnings.warn( 2022-11-23T02:49:35.3522794Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:49:35.3522893Z warnings.warn( 2022-11-23T02:49:35.3522984Z ok (5.565s) 2022-11-23T02:49:35.3522990Z 2022-11-23T02:49:35.3523258Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3523357Z Ran 1 test in 5.565s 2022-11-23T02:49:35.3523363Z 2022-11-23T02:49:35.3523447Z OK 2022-11-23T02:49:35.3523453Z 2022-11-23T02:49:35.3523569Z Generating XML reports... 2022-11-23T02:49:35.3524009Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021714.xml 2022-11-23T02:49:35.3524388Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3524761Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3524928Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3525312Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3525478Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3525496Z 2022-11-23T02:49:35.3525584Z Running tests... 2022-11-23T02:49:35.3525852Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3526166Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52554 2022-11-23T02:49:35.3526418Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52555 2022-11-23T02:49:35.3526678Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3527052Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3527216Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3527596Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3527827Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3528051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3528443Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3528815Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3528980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3529359Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3529535Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3529758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3530148Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3530362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3530574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3530719Z skip: Skipped due to small world size. (4.965s) 2022-11-23T02:49:35.3530728Z 2022-11-23T02:49:35.3530996Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3531084Z Ran 1 test in 4.966s 2022-11-23T02:49:35.3531104Z 2022-11-23T02:49:35.3531189Z OK (skipped=1) 2022-11-23T02:49:35.3531194Z 2022-11-23T02:49:35.3531309Z Generating XML reports... 2022-11-23T02:49:35.3531746Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021723.xml 2022-11-23T02:49:35.3532058Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3532432Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3532598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3533031Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3533324Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3533331Z 2022-11-23T02:49:35.3533446Z Running tests... 2022-11-23T02:49:35.3533767Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3534137Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52761 2022-11-23T02:49:35.3534384Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52762 2022-11-23T02:49:35.3534687Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3535134Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3535332Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3535848Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3536073Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3536338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3536813Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3537262Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3537459Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3537912Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3538117Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3538388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3538861Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3539121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3539517Z STAGE:2022-11-23 02:17:35 52762:52762 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3539774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3540172Z STAGE:2022-11-23 02:17:35 52761:52761 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3540577Z STAGE:2022-11-23 02:17:35 52761:52761 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3540993Z STAGE:2022-11-23 02:17:35 52761:52761 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3541403Z STAGE:2022-11-23 02:17:35 52762:52762 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3541818Z STAGE:2022-11-23 02:17:35 52762:52762 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3542212Z STAGE:2022-11-23 02:17:35 52761:52761 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3542604Z STAGE:2022-11-23 02:17:35 52762:52762 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3543006Z STAGE:2022-11-23 02:17:35 52761:52761 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3543421Z STAGE:2022-11-23 02:17:35 52761:52761 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3543820Z STAGE:2022-11-23 02:17:35 52762:52762 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3544251Z STAGE:2022-11-23 02:17:35 52762:52762 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3544581Z STAGE:2022-11-23 02:17:35 52761:52761 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3544969Z STAGE:2022-11-23 02:17:35 52762:52762 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3545304Z STAGE:2022-11-23 02:17:35 52761:52761 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3545649Z STAGE:2022-11-23 02:17:35 52761:52761 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3545981Z STAGE:2022-11-23 02:17:35 52762:52762 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3546325Z STAGE:2022-11-23 02:17:35 52762:52762 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3547112Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:49:35.3547218Z warnings.warn( 2022-11-23T02:49:35.3547956Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:49:35.3548055Z warnings.warn( 2022-11-23T02:49:35.3548135Z ok (5.069s) 2022-11-23T02:49:35.3548141Z 2022-11-23T02:49:35.3548407Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3548509Z Ran 1 test in 5.070s 2022-11-23T02:49:35.3548516Z 2022-11-23T02:49:35.3548599Z OK 2022-11-23T02:49:35.3548605Z 2022-11-23T02:49:35.3548717Z Generating XML reports... 2022-11-23T02:49:35.3549156Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021733.xml 2022-11-23T02:49:35.3549472Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3549846Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3550009Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3550387Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3550566Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3550571Z 2022-11-23T02:49:35.3550671Z Running tests... 2022-11-23T02:49:35.3550935Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3551255Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52974 2022-11-23T02:49:35.3551462Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52975 2022-11-23T02:49:35.3551722Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3552093Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3552258Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3552635Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3552812Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3553034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3553403Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3553555Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3554003Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3554179Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3554401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3554794Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3555184Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3555398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3555728Z STAGE:2022-11-23 02:17:44 52975:52975 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3555940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3556317Z STAGE:2022-11-23 02:17:45 52974:52974 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3556658Z STAGE:2022-11-23 02:17:45 52975:52975 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3556993Z STAGE:2022-11-23 02:17:45 52974:52974 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3557339Z STAGE:2022-11-23 02:17:45 52975:52975 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3557685Z STAGE:2022-11-23 02:17:45 52974:52974 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3558426Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:49:35.3558530Z warnings.warn( 2022-11-23T02:49:35.3559267Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:49:35.3559366Z warnings.warn( 2022-11-23T02:49:35.3559458Z ok (5.261s) 2022-11-23T02:49:35.3559464Z 2022-11-23T02:49:35.3559730Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3559829Z Ran 1 test in 5.262s 2022-11-23T02:49:35.3559835Z 2022-11-23T02:49:35.3559918Z OK 2022-11-23T02:49:35.3559924Z 2022-11-23T02:49:35.3560039Z Generating XML reports... 2022-11-23T02:49:35.3560478Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021742.xml 2022-11-23T02:49:35.3560792Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3561160Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3561322Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3561704Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3561881Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3561887Z 2022-11-23T02:49:35.3561986Z Running tests... 2022-11-23T02:49:35.3562256Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3562557Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53187 2022-11-23T02:49:35.3562761Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53188 2022-11-23T02:49:35.3563017Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3563445Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3563607Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3563987Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3564163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3564387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3564758Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3564921Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3565343Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3565526Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3565756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3566153Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3566543Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3566754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3567082Z STAGE:2022-11-23 02:17:54 53188:53188 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3567286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3567619Z STAGE:2022-11-23 02:17:54 53187:53187 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3568028Z STAGE:2022-11-23 02:17:54 53187:53187 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3568599Z STAGE:2022-11-23 02:17:54 53187:53187 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 02:17:54 53188:53188 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3568606Z 2022-11-23T02:49:35.3569008Z STAGE:2022-11-23 02:17:54 53188:53188 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3569404Z STAGE:2022-11-23 02:17:54 53187:53187 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3569795Z STAGE:2022-11-23 02:17:54 53188:53188 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3570197Z STAGE:2022-11-23 02:17:54 53187:53187 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3570595Z STAGE:2022-11-23 02:17:54 53188:53188 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3571015Z STAGE:2022-11-23 02:17:54 53187:53187 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3571429Z STAGE:2022-11-23 02:17:54 53188:53188 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3571539Z ok (5.078s) 2022-11-23T02:49:35.3571546Z 2022-11-23T02:49:35.3571875Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3571995Z Ran 1 test in 5.078s 2022-11-23T02:49:35.3572002Z 2022-11-23T02:49:35.3572098Z OK 2022-11-23T02:49:35.3572105Z 2022-11-23T02:49:35.3572242Z Generating XML reports... 2022-11-23T02:49:35.3572777Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021751.xml 2022-11-23T02:49:35.3573151Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3573686Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3573884Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3574340Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3574547Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3574556Z 2022-11-23T02:49:35.3574676Z Running tests... 2022-11-23T02:49:35.3574994Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3575274Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s) 2022-11-23T02:49:35.3575294Z 2022-11-23T02:49:35.3575597Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3575718Z Ran 1 test in 0.002s 2022-11-23T02:49:35.3575725Z 2022-11-23T02:49:35.3575840Z OK (skipped=1) 2022-11-23T02:49:35.3575906Z 2022-11-23T02:49:35.3576048Z Generating XML reports... 2022-11-23T02:49:35.3576576Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021800.xml 2022-11-23T02:49:35.3576947Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3577398Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3577589Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3578051Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3578264Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3578271Z 2022-11-23T02:49:35.3578387Z Running tests... 2022-11-23T02:49:35.3578703Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3579027Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s) 2022-11-23T02:49:35.3579034Z 2022-11-23T02:49:35.3579351Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3579466Z Ran 1 test in 0.002s 2022-11-23T02:49:35.3579473Z 2022-11-23T02:49:35.3579585Z OK (skipped=1) 2022-11-23T02:49:35.3579592Z 2022-11-23T02:49:35.3579728Z Generating XML reports... 2022-11-23T02:49:35.3580249Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021805.xml 2022-11-23T02:49:35.3580618Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3581067Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3581262Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3581727Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3581932Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3581954Z 2022-11-23T02:49:35.3582062Z Running tests... 2022-11-23T02:49:35.3582381Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3582740Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53532 2022-11-23T02:49:35.3582987Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53533 2022-11-23T02:49:35.3583286Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3583733Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3583996Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3584442Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3584618Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3584843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3585233Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3585601Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3585766Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3586145Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3586398Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3586626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3587020Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3587233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3587446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3587668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:49:35.3587893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:49:35.3588286Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.3588679Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.3588995Z STAGE:2022-11-23 02:18:11 53532:53532 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3589325Z STAGE:2022-11-23 02:18:11 53533:53533 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3589662Z STAGE:2022-11-23 02:18:11 53532:53532 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3590000Z STAGE:2022-11-23 02:18:11 53533:53533 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3590351Z STAGE:2022-11-23 02:18:11 53532:53532 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3590704Z STAGE:2022-11-23 02:18:11 53533:53533 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3591032Z STAGE:2022-11-23 02:18:11 53532:53532 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3591364Z STAGE:2022-11-23 02:18:11 53533:53533 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3591703Z STAGE:2022-11-23 02:18:11 53533:53533 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3592035Z STAGE:2022-11-23 02:18:11 53532:53532 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3592384Z STAGE:2022-11-23 02:18:11 53533:53533 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3592730Z STAGE:2022-11-23 02:18:11 53532:53532 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3592825Z ok (5.461s) 2022-11-23T02:49:35.3592832Z 2022-11-23T02:49:35.3593096Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3593194Z Ran 1 test in 5.462s 2022-11-23T02:49:35.3593200Z 2022-11-23T02:49:35.3593282Z OK 2022-11-23T02:49:35.3593288Z 2022-11-23T02:49:35.3593466Z Generating XML reports... 2022-11-23T02:49:35.3593910Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021809.xml 2022-11-23T02:49:35.3594221Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3594593Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3594758Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3595138Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3595304Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3595323Z 2022-11-23T02:49:35.3595410Z Running tests... 2022-11-23T02:49:35.3595675Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3596023Z test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53751 2022-11-23T02:49:35.3596233Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53752 2022-11-23T02:49:35.3596487Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3596859Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3597023Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3597411Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3597588Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3597810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3598208Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3598578Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3598741Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3599121Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3599299Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3599521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3599912Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3600125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3600343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3600490Z skip: Skipped due to small world size. (4.962s) 2022-11-23T02:49:35.3600496Z 2022-11-23T02:49:35.3600760Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3600861Z Ran 1 test in 4.962s 2022-11-23T02:49:35.3600867Z 2022-11-23T02:49:35.3600951Z OK (skipped=1) 2022-11-23T02:49:35.3600970Z 2022-11-23T02:49:35.3601074Z Generating XML reports... 2022-11-23T02:49:35.3601514Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021818.xml 2022-11-23T02:49:35.3601824Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3602197Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3602359Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3602803Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3602980Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3602986Z 2022-11-23T02:49:35.3603085Z Running tests... 2022-11-23T02:49:35.3603351Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3603632Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s) 2022-11-23T02:49:35.3603638Z 2022-11-23T02:49:35.3603901Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3604009Z Ran 1 test in 0.002s 2022-11-23T02:49:35.3604015Z 2022-11-23T02:49:35.3604111Z OK (skipped=1) 2022-11-23T02:49:35.3604116Z 2022-11-23T02:49:35.3604231Z Generating XML reports... 2022-11-23T02:49:35.3604714Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021827.xml 2022-11-23T02:49:35.3605034Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3605409Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3605572Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3605951Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3606129Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3606135Z 2022-11-23T02:49:35.3606234Z Running tests... 2022-11-23T02:49:35.3606497Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3606769Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s) 2022-11-23T02:49:35.3606792Z 2022-11-23T02:49:35.3607044Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3607144Z Ran 1 test in 0.002s 2022-11-23T02:49:35.3607150Z 2022-11-23T02:49:35.3607246Z OK (skipped=1) 2022-11-23T02:49:35.3607251Z 2022-11-23T02:49:35.3607366Z Generating XML reports... 2022-11-23T02:49:35.3607923Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021832.xml 2022-11-23T02:49:35.3608239Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3608612Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3608819Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3609268Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3609493Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3609501Z 2022-11-23T02:49:35.3609616Z Running tests... 2022-11-23T02:49:35.3609933Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3610258Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s) 2022-11-23T02:49:35.3610265Z 2022-11-23T02:49:35.3610580Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3610697Z Ran 1 test in 0.002s 2022-11-23T02:49:35.3610704Z 2022-11-23T02:49:35.3610823Z OK (skipped=1) 2022-11-23T02:49:35.3610829Z 2022-11-23T02:49:35.3610962Z Generating XML reports... 2022-11-23T02:49:35.3611483Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021836.xml 2022-11-23T02:49:35.3611860Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3612398Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3612589Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3613048Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3613252Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3613270Z 2022-11-23T02:49:35.3613374Z Running tests... 2022-11-23T02:49:35.3613696Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3614031Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s) 2022-11-23T02:49:35.3614038Z 2022-11-23T02:49:35.3614356Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3614481Z Ran 1 test in 0.002s 2022-11-23T02:49:35.3614493Z 2022-11-23T02:49:35.3614663Z OK (skipped=1) 2022-11-23T02:49:35.3614671Z 2022-11-23T02:49:35.3614817Z Generating XML reports... 2022-11-23T02:49:35.3615345Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021840.xml 2022-11-23T02:49:35.3615717Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3616162Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3616361Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3616815Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3617026Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3617033Z 2022-11-23T02:49:35.3617150Z Running tests... 2022-11-23T02:49:35.3617475Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3617854Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54222 2022-11-23T02:49:35.3618101Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54223 2022-11-23T02:49:35.3618404Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3618849Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3619042Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3619495Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3619709Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3619973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3620449Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3620892Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3621089Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3621548Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3621759Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3622024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3622493Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3622822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3623077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3623184Z ok (4.966s) 2022-11-23T02:49:35.3623190Z 2022-11-23T02:49:35.3623512Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3623631Z Ran 1 test in 4.966s 2022-11-23T02:49:35.3623638Z 2022-11-23T02:49:35.3623735Z OK 2022-11-23T02:49:35.3623743Z 2022-11-23T02:49:35.3623880Z Generating XML reports... 2022-11-23T02:49:35.3624418Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021844.xml 2022-11-23T02:49:35.3624726Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3625098Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3625326Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3625714Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3625891Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3625896Z 2022-11-23T02:49:35.3625995Z Running tests... 2022-11-23T02:49:35.3626248Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3626561Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54429 2022-11-23T02:49:35.3626767Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54430 2022-11-23T02:49:35.3627023Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3627396Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3627560Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3627941Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3628117Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3628341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3628733Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3629103Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3629268Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3629648Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3629828Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3630052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3630440Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3630655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3630871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3631094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:49:35.3631315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:49:35.3631709Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.3632158Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.3632379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T02:49:35.3632601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T02:49:35.3632977Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T02:49:35.3633366Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T02:49:35.3633584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T02:49:35.3633844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T02:49:35.3634242Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T02:49:35.3634630Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T02:49:35.3634721Z ok (5.070s) 2022-11-23T02:49:35.3634728Z 2022-11-23T02:49:35.3634997Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3635098Z Ran 1 test in 5.070s 2022-11-23T02:49:35.3635104Z 2022-11-23T02:49:35.3635190Z OK 2022-11-23T02:49:35.3635196Z 2022-11-23T02:49:35.3635309Z Generating XML reports... 2022-11-23T02:49:35.3635749Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021853.xml 2022-11-23T02:49:35.3636059Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3636435Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3636598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3636978Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3637154Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3637160Z 2022-11-23T02:49:35.3637259Z Running tests... 2022-11-23T02:49:35.3637524Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3637764Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports all_gather_v (0.002s) 2022-11-23T02:49:35.3637770Z 2022-11-23T02:49:35.3638032Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3638131Z Ran 1 test in 0.003s 2022-11-23T02:49:35.3638137Z 2022-11-23T02:49:35.3638234Z OK (skipped=1) 2022-11-23T02:49:35.3638245Z 2022-11-23T02:49:35.3638348Z Generating XML reports... 2022-11-23T02:49:35.3638785Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021902.xml 2022-11-23T02:49:35.3639095Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3639465Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3639628Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3640007Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3640181Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3640187Z 2022-11-23T02:49:35.3640286Z Running tests... 2022-11-23T02:49:35.3640551Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3640930Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54726 2022-11-23T02:49:35.3641138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54727 2022-11-23T02:49:35.3641392Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3641764Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3641927Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3642304Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3642480Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3642817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3643196Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3643358Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3643742Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3643921Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3644144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3644535Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3644913Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3645132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3645348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3645568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:49:35.3645789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:49:35.3646179Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.3646565Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.3646896Z STAGE:2022-11-23 02:19:09 54726:54726 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3647227Z STAGE:2022-11-23 02:19:09 54727:54727 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3647569Z STAGE:2022-11-23 02:19:09 54726:54726 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3647947Z STAGE:2022-11-23 02:19:09 54727:54727 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3648301Z STAGE:2022-11-23 02:19:09 54726:54726 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3648661Z STAGE:2022-11-23 02:19:09 54727:54727 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3649057Z STAGE:2022-11-23 02:19:09 54726:54726 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3649441Z STAGE:2022-11-23 02:19:09 54727:54727 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3649835Z STAGE:2022-11-23 02:19:09 54726:54726 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3650236Z STAGE:2022-11-23 02:19:09 54727:54727 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3650738Z STAGE:2022-11-23 02:19:09 54727:54727 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3651158Z STAGE:2022-11-23 02:19:09 54726:54726 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3652056Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:49:35.3652176Z warnings.warn( 2022-11-23T02:49:35.3653064Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:49:35.3653181Z warnings.warn( 2022-11-23T02:49:35.3653297Z ok (5.179s) 2022-11-23T02:49:35.3653365Z 2022-11-23T02:49:35.3653692Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3653810Z Ran 1 test in 5.179s 2022-11-23T02:49:35.3653816Z 2022-11-23T02:49:35.3653901Z OK 2022-11-23T02:49:35.3653918Z 2022-11-23T02:49:35.3654041Z Generating XML reports... 2022-11-23T02:49:35.3654569Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021906.xml 2022-11-23T02:49:35.3654945Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3655388Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3655582Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3656041Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3656265Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3656272Z 2022-11-23T02:49:35.3656386Z Running tests... 2022-11-23T02:49:35.3656703Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3657104Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54945 2022-11-23T02:49:35.3657345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54946 2022-11-23T02:49:35.3657651Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3658095Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3658292Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3658751Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3658967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3659234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3659680Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3659875Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3660330Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3660545Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3660815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3661278Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3661832Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3662092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3662345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3662609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:49:35.3662871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:49:35.3663346Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.3663816Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.3664267Z STAGE:2022-11-23 02:19:19 54945:54945 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3664670Z STAGE:2022-11-23 02:19:19 54946:54946 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3665072Z STAGE:2022-11-23 02:19:19 54945:54945 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3665481Z STAGE:2022-11-23 02:19:19 54946:54946 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3665901Z STAGE:2022-11-23 02:19:19 54945:54945 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3666317Z STAGE:2022-11-23 02:19:19 54946:54946 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3666709Z STAGE:2022-11-23 02:19:19 54946:54946 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3667104Z STAGE:2022-11-23 02:19:19 54945:54945 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3667511Z STAGE:2022-11-23 02:19:19 54945:54945 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3667931Z STAGE:2022-11-23 02:19:19 54945:54945 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3668328Z STAGE:2022-11-23 02:19:19 54946:54946 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3668745Z STAGE:2022-11-23 02:19:19 54946:54946 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3669640Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:49:35.3669757Z warnings.warn( 2022-11-23T02:49:35.3670651Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:49:35.3670780Z warnings.warn( 2022-11-23T02:49:35.3670886Z ok (5.170s) 2022-11-23T02:49:35.3670893Z 2022-11-23T02:49:35.3671211Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3671321Z Ran 1 test in 5.170s 2022-11-23T02:49:35.3671340Z 2022-11-23T02:49:35.3671425Z OK 2022-11-23T02:49:35.3671431Z 2022-11-23T02:49:35.3671569Z Generating XML reports... 2022-11-23T02:49:35.3672094Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021916.xml 2022-11-23T02:49:35.3672468Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3672910Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3673170Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3673633Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3673848Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3673855Z 2022-11-23T02:49:35.3673972Z Running tests... 2022-11-23T02:49:35.3674295Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3674685Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55164 2022-11-23T02:49:35.3674890Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55165 2022-11-23T02:49:35.3675141Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3675554Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3675726Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3676112Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3676288Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3676513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3676882Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3677045Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3677423Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3677603Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3677817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3678211Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3678597Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3678810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3679023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3679242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:49:35.3679461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:49:35.3679851Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.3680239Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.3680567Z STAGE:2022-11-23 02:19:28 55164:55164 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3680898Z STAGE:2022-11-23 02:19:28 55165:55165 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3681231Z STAGE:2022-11-23 02:19:28 55164:55164 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3681581Z STAGE:2022-11-23 02:19:28 55164:55164 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3681913Z STAGE:2022-11-23 02:19:28 55165:55165 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3682260Z STAGE:2022-11-23 02:19:28 55165:55165 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3682646Z STAGE:2022-11-23 02:19:28 55164:55164 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3682973Z STAGE:2022-11-23 02:19:28 55165:55165 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3683308Z STAGE:2022-11-23 02:19:28 55165:55165 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3683639Z STAGE:2022-11-23 02:19:28 55164:55164 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3683989Z STAGE:2022-11-23 02:19:28 55165:55165 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3684335Z STAGE:2022-11-23 02:19:28 55164:55164 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3685150Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:49:35.3685258Z warnings.warn( 2022-11-23T02:49:35.3685996Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:49:35.3686096Z warnings.warn( 2022-11-23T02:49:35.3686188Z ok (4.962s) 2022-11-23T02:49:35.3686194Z 2022-11-23T02:49:35.3686460Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3686548Z Ran 1 test in 4.963s 2022-11-23T02:49:35.3686554Z 2022-11-23T02:49:35.3686638Z OK 2022-11-23T02:49:35.3686643Z 2022-11-23T02:49:35.3686758Z Generating XML reports... 2022-11-23T02:49:35.3687201Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021925.xml 2022-11-23T02:49:35.3687518Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3687925Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3688098Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3688480Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3688660Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3688665Z 2022-11-23T02:49:35.3688765Z Running tests... 2022-11-23T02:49:35.3689031Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3689354Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55383 2022-11-23T02:49:35.3689563Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55384 2022-11-23T02:49:35.3689821Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3690191Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3690353Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3690731Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3690909Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3691132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3691528Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3691975Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3692138Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3692531Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3692713Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3692937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3693329Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3693552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3693764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3694036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:49:35.3694270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:49:35.3694669Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.3695059Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.3695396Z STAGE:2022-11-23 02:19:37 55384:55384 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3695728Z STAGE:2022-11-23 02:19:37 55383:55383 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3696064Z STAGE:2022-11-23 02:19:37 55384:55384 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3696415Z STAGE:2022-11-23 02:19:37 55384:55384 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3696755Z STAGE:2022-11-23 02:19:37 55383:55383 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3697103Z STAGE:2022-11-23 02:19:37 55383:55383 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3697418Z STAGE:2022-11-23 02:19:37 55384:55384 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3697745Z STAGE:2022-11-23 02:19:37 55383:55383 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3698083Z STAGE:2022-11-23 02:19:37 55384:55384 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3698431Z STAGE:2022-11-23 02:19:37 55384:55384 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3698767Z STAGE:2022-11-23 02:19:37 55383:55383 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3699116Z STAGE:2022-11-23 02:19:37 55383:55383 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3699860Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:49:35.3699965Z warnings.warn( 2022-11-23T02:49:35.3700693Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:49:35.3700797Z warnings.warn( 2022-11-23T02:49:35.3700894Z ok (5.442s) 2022-11-23T02:49:35.3700900Z 2022-11-23T02:49:35.3701166Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3701266Z Ran 1 test in 5.443s 2022-11-23T02:49:35.3701327Z 2022-11-23T02:49:35.3701417Z OK 2022-11-23T02:49:35.3701423Z 2022-11-23T02:49:35.3701540Z Generating XML reports... 2022-11-23T02:49:35.3701986Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021934.xml 2022-11-23T02:49:35.3702298Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3702675Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3702841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3703223Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3703402Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3703408Z 2022-11-23T02:49:35.3703512Z Running tests... 2022-11-23T02:49:35.3703823Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3704146Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55602 2022-11-23T02:49:35.3704353Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55603 2022-11-23T02:49:35.3704593Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3704971Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3705135Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3705520Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3705700Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3705930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3706302Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3706467Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3706853Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3707035Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3707263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3707658Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3708051Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3708274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3708490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3708638Z skip: Skipped due to small world size. (5.369s) 2022-11-23T02:49:35.3708645Z 2022-11-23T02:49:35.3708915Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3709019Z Ran 1 test in 5.370s 2022-11-23T02:49:35.3709025Z 2022-11-23T02:49:35.3709122Z OK (skipped=1) 2022-11-23T02:49:35.3709128Z 2022-11-23T02:49:35.3709245Z Generating XML reports... 2022-11-23T02:49:35.3709691Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021944.xml 2022-11-23T02:49:35.3710004Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3710379Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3710592Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3710987Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3711171Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3711177Z 2022-11-23T02:49:35.3711278Z Running tests... 2022-11-23T02:49:35.3711545Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3711869Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55809 2022-11-23T02:49:35.3712079Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55810 2022-11-23T02:49:35.3712336Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3712761Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3712934Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3713321Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3713501Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3713725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3714094Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3714261Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3714647Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3714830Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3715059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3715453Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3715843Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3716057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3716273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3716409Z skip: Skipped due to small world size. (5.403s) 2022-11-23T02:49:35.3716429Z 2022-11-23T02:49:35.3716684Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3716790Z Ran 1 test in 5.403s 2022-11-23T02:49:35.3716796Z 2022-11-23T02:49:35.3716901Z OK (skipped=1) 2022-11-23T02:49:35.3716907Z 2022-11-23T02:49:35.3717027Z Generating XML reports... 2022-11-23T02:49:35.3717467Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021953.xml 2022-11-23T02:49:35.3717779Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3718152Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3718322Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3718704Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3718882Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3718888Z 2022-11-23T02:49:35.3718988Z Running tests... 2022-11-23T02:49:35.3719260Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3719642Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56016 2022-11-23T02:49:35.3719852Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56017 2022-11-23T02:49:35.3720109Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3720489Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3720656Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3721038Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3721218Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3721487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3721871Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3722039Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3722408Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3722585Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3722808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3723201Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3723600Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3723825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3724045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3724195Z skip: Skipped due to small world size. (5.662s) 2022-11-23T02:49:35.3724202Z 2022-11-23T02:49:35.3724472Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3724574Z Ran 1 test in 5.662s 2022-11-23T02:49:35.3724580Z 2022-11-23T02:49:35.3724678Z OK (skipped=1) 2022-11-23T02:49:35.3724684Z 2022-11-23T02:49:35.3724798Z Generating XML reports... 2022-11-23T02:49:35.3725239Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022003.xml 2022-11-23T02:49:35.3725556Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3725929Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3726103Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3726492Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3726670Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3726676Z 2022-11-23T02:49:35.3726778Z Running tests... 2022-11-23T02:49:35.3727046Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3727368Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56223 2022-11-23T02:49:35.3727576Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56224 2022-11-23T02:49:35.3728037Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3728767Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3728936Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3729325Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3729508Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3729735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3730137Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3730507Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3730677Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3731135Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3731330Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3731556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3731954Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3732169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3732384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3732532Z skip: Skipped due to small world size. (4.979s) 2022-11-23T02:49:35.3732538Z 2022-11-23T02:49:35.3732808Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3732910Z Ran 1 test in 4.979s 2022-11-23T02:49:35.3732915Z 2022-11-23T02:49:35.3733020Z OK (skipped=1) 2022-11-23T02:49:35.3733026Z 2022-11-23T02:49:35.3733147Z Generating XML reports... 2022-11-23T02:49:35.3733586Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022013.xml 2022-11-23T02:49:35.3733900Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3734272Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3734445Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3734816Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3734997Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3735002Z 2022-11-23T02:49:35.3735106Z Running tests... 2022-11-23T02:49:35.3735375Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3735691Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56430 2022-11-23T02:49:35.3735899Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56431 2022-11-23T02:49:35.3736153Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3736524Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3736691Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3737075Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3737255Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3737486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3737941Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3738311Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3738478Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3738863Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3739041Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3739263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3739658Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3739926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3740272Z STAGE:2022-11-23 02:20:25 56431:56431 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3740489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3740822Z STAGE:2022-11-23 02:20:25 56430:56430 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3741148Z STAGE:2022-11-23 02:20:25 56430:56430 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3741499Z STAGE:2022-11-23 02:20:25 56430:56430 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3741834Z STAGE:2022-11-23 02:20:25 56431:56431 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3742186Z STAGE:2022-11-23 02:20:25 56431:56431 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3742519Z STAGE:2022-11-23 02:20:25 56430:56430 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3742851Z STAGE:2022-11-23 02:20:25 56431:56431 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3743191Z STAGE:2022-11-23 02:20:25 56430:56430 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3743544Z STAGE:2022-11-23 02:20:25 56430:56430 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3743882Z STAGE:2022-11-23 02:20:25 56431:56431 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3744232Z STAGE:2022-11-23 02:20:25 56431:56431 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3744982Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:49:35.3745087Z warnings.warn( 2022-11-23T02:49:35.3745938Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:49:35.3746057Z warnings.warn( 2022-11-23T02:49:35.3746166Z ok (5.018s) 2022-11-23T02:49:35.3746173Z 2022-11-23T02:49:35.3746492Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3746610Z Ran 1 test in 5.018s 2022-11-23T02:49:35.3746617Z 2022-11-23T02:49:35.3746727Z OK 2022-11-23T02:49:35.3746734Z 2022-11-23T02:49:35.3746870Z Generating XML reports... 2022-11-23T02:49:35.3747398Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022022.xml 2022-11-23T02:49:35.3747777Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3748293Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3748490Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3748957Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3749173Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3749180Z 2022-11-23T02:49:35.3749284Z Running tests... 2022-11-23T02:49:35.3749604Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3750020Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56643 2022-11-23T02:49:35.3750325Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56644 2022-11-23T02:49:35.3750638Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3751091Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3751291Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3751750Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3751969Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3752240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3752685Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3752883Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3753345Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3753560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3753830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3754307Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3754782Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3755001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3755216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3755966Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:49:35.3756072Z warnings.warn( 2022-11-23T02:49:35.3756808Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:49:35.3756911Z warnings.warn( 2022-11-23T02:49:35.3757006Z ok (5.674s) 2022-11-23T02:49:35.3757012Z 2022-11-23T02:49:35.3757281Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3757368Z Ran 1 test in 5.674s 2022-11-23T02:49:35.3757388Z 2022-11-23T02:49:35.3757459Z OK 2022-11-23T02:49:35.3757464Z 2022-11-23T02:49:35.3757583Z Generating XML reports... 2022-11-23T02:49:35.3758083Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022031.xml 2022-11-23T02:49:35.3758396Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3758772Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3758938Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3759320Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3759497Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3759503Z 2022-11-23T02:49:35.3759604Z Running tests... 2022-11-23T02:49:35.3759872Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3760233Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56850 2022-11-23T02:49:35.3760449Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56851 2022-11-23T02:49:35.3760705Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3761081Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3761255Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3761638Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3761816Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3762041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3762440Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3762813Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3762979Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3763361Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3763526Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3763746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3764138Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3764354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3764691Z STAGE:2022-11-23 02:20:44 56851:56851 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3764909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3765241Z STAGE:2022-11-23 02:20:44 56850:56850 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3765581Z STAGE:2022-11-23 02:20:44 56851:56851 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3765935Z STAGE:2022-11-23 02:20:44 56851:56851 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3766270Z STAGE:2022-11-23 02:20:44 56850:56850 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3766622Z STAGE:2022-11-23 02:20:44 56850:56850 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3766953Z STAGE:2022-11-23 02:20:44 56851:56851 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3767291Z STAGE:2022-11-23 02:20:44 56850:56850 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3767743Z STAGE:2022-11-23 02:20:44 56850:56850 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3768096Z STAGE:2022-11-23 02:20:44 56850:56850 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3768433Z STAGE:2022-11-23 02:20:44 56851:56851 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3768780Z STAGE:2022-11-23 02:20:44 56851:56851 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3769525Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:49:35.3769628Z warnings.warn( 2022-11-23T02:49:35.3770420Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:49:35.3770532Z warnings.warn( 2022-11-23T02:49:35.3770625Z ok (5.119s) 2022-11-23T02:49:35.3770631Z 2022-11-23T02:49:35.3770907Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3771009Z Ran 1 test in 5.119s 2022-11-23T02:49:35.3771015Z 2022-11-23T02:49:35.3771101Z OK 2022-11-23T02:49:35.3771106Z 2022-11-23T02:49:35.3771211Z Generating XML reports... 2022-11-23T02:49:35.3771656Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022041.xml 2022-11-23T02:49:35.3771971Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3772349Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3772518Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3772902Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3773085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3773090Z 2022-11-23T02:49:35.3773192Z Running tests... 2022-11-23T02:49:35.3773460Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3773780Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57063 2022-11-23T02:49:35.3773987Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57064 2022-11-23T02:49:35.3774251Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3774627Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3774795Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3775180Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3775359Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3775584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3775954Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3776118Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3776497Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3776738Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3776965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3777353Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3777747Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3777964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3778179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3778510Z STAGE:2022-11-23 02:20:53 57063:57063 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3778844Z STAGE:2022-11-23 02:20:53 57064:57064 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3779229Z STAGE:2022-11-23 02:20:53 57063:57063 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3779800Z STAGE:2022-11-23 02:20:53 57064:57064 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:20:53 57063:57063 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3779807Z 2022-11-23T02:49:35.3780159Z STAGE:2022-11-23 02:20:53 57064:57064 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3780486Z STAGE:2022-11-23 02:20:53 57063:57063 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3780815Z STAGE:2022-11-23 02:20:53 57064:57064 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3781151Z STAGE:2022-11-23 02:20:53 57064:57064 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3781718Z STAGE:2022-11-23 02:20:53 57063:57063 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:20:53 57064:57064 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3781727Z 2022-11-23T02:49:35.3782080Z STAGE:2022-11-23 02:20:53 57063:57063 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3782818Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:49:35.3782923Z warnings.warn( 2022-11-23T02:49:35.3783653Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:49:35.3783759Z warnings.warn( 2022-11-23T02:49:35.3783856Z ok (4.920s) 2022-11-23T02:49:35.3783862Z 2022-11-23T02:49:35.3784131Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3784233Z Ran 1 test in 4.921s 2022-11-23T02:49:35.3784239Z 2022-11-23T02:49:35.3784325Z OK 2022-11-23T02:49:35.3784330Z 2022-11-23T02:49:35.3784448Z Generating XML reports... 2022-11-23T02:49:35.3784890Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022050.xml 2022-11-23T02:49:35.3785205Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3785581Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3785746Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3786130Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3786362Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3786368Z 2022-11-23T02:49:35.3786473Z Running tests... 2022-11-23T02:49:35.3786729Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3787041Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57276 2022-11-23T02:49:35.3787249Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57277 2022-11-23T02:49:35.3787505Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3787875Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3788041Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3788471Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3788655Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3788883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3789261Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3789427Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3789808Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3789990Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3790214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3790614Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3791007Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3791223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3791554Z STAGE:2022-11-23 02:21:02 57277:57277 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3791771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3792100Z STAGE:2022-11-23 02:21:02 57276:57276 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3792438Z STAGE:2022-11-23 02:21:02 57277:57277 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3792796Z STAGE:2022-11-23 02:21:02 57277:57277 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3793138Z STAGE:2022-11-23 02:21:02 57276:57276 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3793475Z STAGE:2022-11-23 02:21:02 57276:57276 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3793804Z STAGE:2022-11-23 02:21:02 57277:57277 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3794133Z STAGE:2022-11-23 02:21:02 57276:57276 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3794472Z STAGE:2022-11-23 02:21:02 57276:57276 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3794825Z STAGE:2022-11-23 02:21:02 57276:57276 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3795159Z STAGE:2022-11-23 02:21:02 57277:57277 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3795508Z STAGE:2022-11-23 02:21:02 57277:57277 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3796311Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:49:35.3796415Z warnings.warn( 2022-11-23T02:49:35.3797153Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T02:49:35.3797255Z warnings.warn( 2022-11-23T02:49:35.3797349Z ok (5.530s) 2022-11-23T02:49:35.3797355Z 2022-11-23T02:49:35.3797628Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3797731Z Ran 1 test in 5.531s 2022-11-23T02:49:35.3797737Z 2022-11-23T02:49:35.3797826Z OK 2022-11-23T02:49:35.3797835Z 2022-11-23T02:49:35.3797996Z Generating XML reports... 2022-11-23T02:49:35.3798442Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022059.xml 2022-11-23T02:49:35.3798759Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3799135Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3799301Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3799686Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3799870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3799876Z 2022-11-23T02:49:35.3799976Z Running tests... 2022-11-23T02:49:35.3800245Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3800583Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57489 2022-11-23T02:49:35.3800778Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57490 2022-11-23T02:49:35.3801034Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3801407Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3801571Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3801957Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3802135Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3802362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3802735Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3802901Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3803284Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3803467Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3803690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3804088Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3804482Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3804707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3804992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3805087Z ok (5.119s) 2022-11-23T02:49:35.3805093Z 2022-11-23T02:49:35.3805365Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3805466Z Ran 1 test in 5.119s 2022-11-23T02:49:35.3805472Z 2022-11-23T02:49:35.3805556Z OK 2022-11-23T02:49:35.3805561Z 2022-11-23T02:49:35.3805682Z Generating XML reports... 2022-11-23T02:49:35.3806123Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022109.xml 2022-11-23T02:49:35.3806422Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3806795Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3806965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3807397Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3807580Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3807586Z 2022-11-23T02:49:35.3807743Z Running tests... 2022-11-23T02:49:35.3808019Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3808329Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57696 2022-11-23T02:49:35.3808538Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57697 2022-11-23T02:49:35.3808838Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3809278Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3809481Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3809930Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3810143Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3810411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3810851Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3811052Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3811509Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3811722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3812000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3812477Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3812948Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3813203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3813446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3813718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:49:35.3813986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:49:35.3814457Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.3815033Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.3815429Z STAGE:2022-11-23 02:21:21 57696:57696 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3815828Z STAGE:2022-11-23 02:21:21 57697:57697 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3816484Z STAGE:2022-11-23 02:21:21 57697:57697 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:21:21 57696:57696 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3816492Z 2022-11-23T02:49:35.3817189Z STAGE:2022-11-23 02:21:21 57697:57697 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 02:21:21 57696:57696 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3817197Z 2022-11-23T02:49:35.3817650Z STAGE:2022-11-23 02:21:21 57696:57696 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3818064Z STAGE:2022-11-23 02:21:21 57697:57697 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3818714Z STAGE:2022-11-23 02:21:21 57697:57697 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:21:21 57696:57696 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3818722Z 2022-11-23T02:49:35.3819151Z STAGE:2022-11-23 02:21:21 57696:57696 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3819571Z STAGE:2022-11-23 02:21:21 57697:57697 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3819683Z ok (5.017s) 2022-11-23T02:49:35.3819690Z 2022-11-23T02:49:35.3820013Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3820132Z Ran 1 test in 5.018s 2022-11-23T02:49:35.3820139Z 2022-11-23T02:49:35.3820243Z OK 2022-11-23T02:49:35.3820254Z 2022-11-23T02:49:35.3820400Z Generating XML reports... 2022-11-23T02:49:35.3820933Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022118.xml 2022-11-23T02:49:35.3821313Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3821765Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3821966Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3822430Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3822647Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3822654Z 2022-11-23T02:49:35.3822773Z Running tests... 2022-11-23T02:49:35.3823095Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3823475Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57915 2022-11-23T02:49:35.3823731Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57916 2022-11-23T02:49:35.3824043Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3824476Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3824684Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3825085Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3825266Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3825492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3825929Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3826095Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3826480Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3826662Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3826888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3827283Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3827676Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3827931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3828155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3828381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:49:35.3828603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:49:35.3828998Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.3829390Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.3829724Z STAGE:2022-11-23 02:21:31 57915:57915 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3830057Z STAGE:2022-11-23 02:21:31 57916:57916 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3830397Z STAGE:2022-11-23 02:21:31 57915:57915 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3830736Z STAGE:2022-11-23 02:21:31 57916:57916 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3831087Z STAGE:2022-11-23 02:21:31 57915:57915 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3831439Z STAGE:2022-11-23 02:21:31 57916:57916 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3831755Z STAGE:2022-11-23 02:21:31 57915:57915 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3832091Z STAGE:2022-11-23 02:21:31 57916:57916 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3832639Z STAGE:2022-11-23 02:21:31 57915:57915 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:21:31 57916:57916 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3832661Z 2022-11-23T02:49:35.3833223Z STAGE:2022-11-23 02:21:31 57915:57915 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 02:21:31 57916:57916 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3833243Z 2022-11-23T02:49:35.3833324Z ok (5.524s) 2022-11-23T02:49:35.3833343Z 2022-11-23T02:49:35.3833598Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3833701Z Ran 1 test in 5.525s 2022-11-23T02:49:35.3833707Z 2022-11-23T02:49:35.3833794Z OK 2022-11-23T02:49:35.3833800Z 2022-11-23T02:49:35.3833915Z Generating XML reports... 2022-11-23T02:49:35.3834356Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022128.xml 2022-11-23T02:49:35.3834668Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3835039Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3835268Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3835658Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3835838Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3835845Z 2022-11-23T02:49:35.3835948Z Running tests... 2022-11-23T02:49:35.3836216Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3836537Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58134 2022-11-23T02:49:35.3836747Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58135 2022-11-23T02:49:35.3837000Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3837418Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3837593Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3837982Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3838161Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3838388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3838763Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3838916Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3839299Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3839481Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3839708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3840102Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3840493Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3840710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3840923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3841147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:49:35.3841371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:49:35.3841767Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.3842163Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.3842494Z STAGE:2022-11-23 02:21:40 58134:58134 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3842821Z STAGE:2022-11-23 02:21:40 58135:58135 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3843158Z STAGE:2022-11-23 02:21:41 58134:58134 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3843511Z STAGE:2022-11-23 02:21:41 58134:58134 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3843846Z STAGE:2022-11-23 02:21:41 58135:58135 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3844195Z STAGE:2022-11-23 02:21:41 58135:58135 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3844582Z STAGE:2022-11-23 02:21:41 58134:58134 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3844910Z STAGE:2022-11-23 02:21:41 58135:58135 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3845246Z STAGE:2022-11-23 02:21:41 58134:58134 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3845583Z STAGE:2022-11-23 02:21:41 58135:58135 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3845932Z STAGE:2022-11-23 02:21:41 58134:58134 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3846284Z STAGE:2022-11-23 02:21:41 58135:58135 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3846368Z ok (5.018s) 2022-11-23T02:49:35.3846391Z 2022-11-23T02:49:35.3846645Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3846748Z Ran 1 test in 5.019s 2022-11-23T02:49:35.3846802Z 2022-11-23T02:49:35.3846892Z OK 2022-11-23T02:49:35.3846897Z 2022-11-23T02:49:35.3847016Z Generating XML reports... 2022-11-23T02:49:35.3847460Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022138.xml 2022-11-23T02:49:35.3847884Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3848262Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3848431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3848814Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3848994Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3849000Z 2022-11-23T02:49:35.3849100Z Running tests... 2022-11-23T02:49:35.3849374Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3849685Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58353 2022-11-23T02:49:35.3849893Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58354 2022-11-23T02:49:35.3850146Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3850520Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3850686Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3851071Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3851248Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3851480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3851877Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3852248Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3852401Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3852781Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3852960Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3853185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3853581Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3853872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3854089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3854312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:49:35.3854534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:49:35.3854930Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.3855266Z STAGE:2022-11-23 02:21:50 58353:58353 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3855660Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.3856038Z STAGE:2022-11-23 02:21:50 58354:58354 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3856385Z STAGE:2022-11-23 02:21:50 58354:58354 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3856743Z STAGE:2022-11-23 02:21:50 58354:58354 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3857080Z STAGE:2022-11-23 02:21:50 58353:58353 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3857431Z STAGE:2022-11-23 02:21:50 58353:58353 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3857760Z STAGE:2022-11-23 02:21:50 58354:58354 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3858088Z STAGE:2022-11-23 02:21:50 58353:58353 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3858426Z STAGE:2022-11-23 02:21:50 58354:58354 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3858780Z STAGE:2022-11-23 02:21:50 58354:58354 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3859116Z STAGE:2022-11-23 02:21:50 58353:58353 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3859462Z STAGE:2022-11-23 02:21:50 58353:58353 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3859560Z ok (5.017s) 2022-11-23T02:49:35.3859566Z 2022-11-23T02:49:35.3859822Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3859924Z Ran 1 test in 5.017s 2022-11-23T02:49:35.3859930Z 2022-11-23T02:49:35.3860016Z OK 2022-11-23T02:49:35.3860021Z 2022-11-23T02:49:35.3860143Z Generating XML reports... 2022-11-23T02:49:35.3860582Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022147.xml 2022-11-23T02:49:35.3860893Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3861270Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3861442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3861823Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3862006Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3862012Z 2022-11-23T02:49:35.3862113Z Running tests... 2022-11-23T02:49:35.3862383Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3862694Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58572 2022-11-23T02:49:35.3862903Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58573 2022-11-23T02:49:35.3863165Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3863609Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3863774Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3864157Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3864338Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3864560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3864929Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3865097Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3865464Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3865689Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3865915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3866316Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3866717Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3866935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3867152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3867302Z skip: Skipped due to small world size. (4.827s) 2022-11-23T02:49:35.3867308Z 2022-11-23T02:49:35.3867579Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3867686Z Ran 1 test in 4.827s 2022-11-23T02:49:35.3867698Z 2022-11-23T02:49:35.3867798Z OK (skipped=1) 2022-11-23T02:49:35.3867804Z 2022-11-23T02:49:35.3867921Z Generating XML reports... 2022-11-23T02:49:35.3868362Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022156.xml 2022-11-23T02:49:35.3868677Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3869050Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3869218Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3869601Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3869784Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3869790Z 2022-11-23T02:49:35.3869892Z Running tests... 2022-11-23T02:49:35.3870167Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3870474Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58779 2022-11-23T02:49:35.3870685Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58780 2022-11-23T02:49:35.3870944Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3871303Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3871467Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3871848Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3872029Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3872340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3872736Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3873110Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3873275Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3873659Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3873841Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3874066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3874460Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3874739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3874955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3875107Z skip: Skipped due to small world size. (4.929s) 2022-11-23T02:49:35.3875113Z 2022-11-23T02:49:35.3875385Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3875487Z Ran 1 test in 4.929s 2022-11-23T02:49:35.3875493Z 2022-11-23T02:49:35.3875593Z OK (skipped=1) 2022-11-23T02:49:35.3875599Z 2022-11-23T02:49:35.3875716Z Generating XML reports... 2022-11-23T02:49:35.3876161Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022205.xml 2022-11-23T02:49:35.3876476Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3876859Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3877027Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3877398Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3877579Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3877585Z 2022-11-23T02:49:35.3877689Z Running tests... 2022-11-23T02:49:35.3877958Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3878271Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58986 2022-11-23T02:49:35.3878476Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58987 2022-11-23T02:49:35.3878734Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3879115Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3879279Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3879662Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3879842Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3880068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3880463Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3880836Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3881003Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3881447Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3881633Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3881857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3882265Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3882485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3882702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3882852Z skip: Skipped due to small world size. (4.817s) 2022-11-23T02:49:35.3882862Z 2022-11-23T02:49:35.3883129Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3883218Z Ran 1 test in 4.817s 2022-11-23T02:49:35.3883273Z 2022-11-23T02:49:35.3883374Z OK (skipped=1) 2022-11-23T02:49:35.3883380Z 2022-11-23T02:49:35.3883501Z Generating XML reports... 2022-11-23T02:49:35.3883941Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022214.xml 2022-11-23T02:49:35.3884254Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3884628Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3884793Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3885176Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3885356Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3885362Z 2022-11-23T02:49:35.3885462Z Running tests... 2022-11-23T02:49:35.3885738Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3886050Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59193 2022-11-23T02:49:35.3886258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59194 2022-11-23T02:49:35.3886512Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3886887Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3887051Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3887436Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3887614Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3887898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3888272Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3888438Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3888819Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3888985Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3889208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3889602Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3889994Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3890282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3890496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3890647Z skip: Skipped due to small world size. (4.828s) 2022-11-23T02:49:35.3890653Z 2022-11-23T02:49:35.3890924Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3891030Z Ran 1 test in 4.829s 2022-11-23T02:49:35.3891036Z 2022-11-23T02:49:35.3891135Z OK (skipped=1) 2022-11-23T02:49:35.3891141Z 2022-11-23T02:49:35.3891261Z Generating XML reports... 2022-11-23T02:49:35.3891703Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022223.xml 2022-11-23T02:49:35.3892017Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3892438Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3892614Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3893004Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3893183Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3893189Z 2022-11-23T02:49:35.3893290Z Running tests... 2022-11-23T02:49:35.3893558Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3893854Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59400 2022-11-23T02:49:35.3894061Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59401 2022-11-23T02:49:35.3894316Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3894697Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3894849Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3895232Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3895414Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3895639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3896009Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3896174Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3896556Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3896739Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3896966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3897364Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3937027Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3937542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3937768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3938406Z STAGE:2022-11-23 02:22:35 59400:59400 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3938740Z STAGE:2022-11-23 02:22:35 59401:59401 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3939531Z STAGE:2022-11-23 02:22:35 59401:59401 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3939871Z STAGE:2022-11-23 02:22:35 59400:59400 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3940225Z STAGE:2022-11-23 02:22:35 59401:59401 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3940579Z STAGE:2022-11-23 02:22:35 59400:59400 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3940910Z STAGE:2022-11-23 02:22:35 59401:59401 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3941243Z STAGE:2022-11-23 02:22:35 59400:59400 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3941571Z STAGE:2022-11-23 02:22:35 59401:59401 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3941904Z STAGE:2022-11-23 02:22:35 59400:59400 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3942376Z STAGE:2022-11-23 02:22:35 59401:59401 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3942736Z STAGE:2022-11-23 02:22:35 59400:59400 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3942828Z ok (5.220s) 2022-11-23T02:49:35.3942836Z 2022-11-23T02:49:35.3943110Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3943216Z Ran 1 test in 5.221s 2022-11-23T02:49:35.3943223Z 2022-11-23T02:49:35.3943306Z OK 2022-11-23T02:49:35.3943312Z 2022-11-23T02:49:35.3943427Z Generating XML reports... 2022-11-23T02:49:35.3943875Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022232.xml 2022-11-23T02:49:35.3944196Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3944579Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3944750Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3945137Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3945318Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3945324Z 2022-11-23T02:49:35.3945421Z Running tests... 2022-11-23T02:49:35.3945690Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3945987Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59613 2022-11-23T02:49:35.3946195Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59614 2022-11-23T02:49:35.3946450Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3946828Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3946996Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3947367Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3947546Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3947773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3948173Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3948544Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3948707Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3949097Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3949335Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3949564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3949965Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3950179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3950510Z STAGE:2022-11-23 02:22:44 59614:59614 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3950724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3951053Z STAGE:2022-11-23 02:22:44 59613:59613 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3951438Z STAGE:2022-11-23 02:22:44 59614:59614 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3951799Z STAGE:2022-11-23 02:22:44 59614:59614 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3952139Z STAGE:2022-11-23 02:22:44 59613:59613 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3952491Z STAGE:2022-11-23 02:22:44 59613:59613 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3952818Z STAGE:2022-11-23 02:22:44 59614:59614 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3953142Z STAGE:2022-11-23 02:22:44 59613:59613 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3953480Z STAGE:2022-11-23 02:22:44 59614:59614 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3953831Z STAGE:2022-11-23 02:22:44 59614:59614 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3954172Z STAGE:2022-11-23 02:22:44 59613:59613 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3954522Z STAGE:2022-11-23 02:22:44 59613:59613 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3954604Z ok (5.018s) 2022-11-23T02:49:35.3954622Z 2022-11-23T02:49:35.3954879Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3954980Z Ran 1 test in 5.018s 2022-11-23T02:49:35.3954987Z 2022-11-23T02:49:35.3955071Z OK 2022-11-23T02:49:35.3955077Z 2022-11-23T02:49:35.3955191Z Generating XML reports... 2022-11-23T02:49:35.3955635Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022242.xml 2022-11-23T02:49:35.3955950Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3956321Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3956492Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3956878Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3957057Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3957063Z 2022-11-23T02:49:35.3957161Z Running tests... 2022-11-23T02:49:35.3957433Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3957737Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59826 2022-11-23T02:49:35.3957943Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59827 2022-11-23T02:49:35.3958198Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3958574Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3958797Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3959184Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3959362Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3959585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3959959Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3960111Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3960496Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3960674Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3960946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3961350Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3961745Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3961957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3962168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3962499Z STAGE:2022-11-23 02:22:54 59827:59827 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3962830Z STAGE:2022-11-23 02:22:54 59826:59826 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3963170Z STAGE:2022-11-23 02:22:54 59826:59826 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3963506Z STAGE:2022-11-23 02:22:54 59827:59827 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3963857Z STAGE:2022-11-23 02:22:54 59826:59826 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3964080Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:49:35.3964428Z STAGE:2022-11-23 02:22:54 59827:59827 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3964652Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:49:35.3964987Z STAGE:2022-11-23 02:22:54 59826:59826 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3965313Z STAGE:2022-11-23 02:22:54 59827:59827 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3965649Z STAGE:2022-11-23 02:22:54 59826:59826 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3965990Z STAGE:2022-11-23 02:22:54 59827:59827 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3966337Z STAGE:2022-11-23 02:22:54 59826:59826 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3966685Z STAGE:2022-11-23 02:22:54 59827:59827 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3967482Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1506: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2022-11-23T02:49:35.3967584Z warnings.warn( 2022-11-23T02:49:35.3968461Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1506: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2022-11-23T02:49:35.3968627Z warnings.warn( 2022-11-23T02:49:35.3968708Z ok (5.824s) 2022-11-23T02:49:35.3968726Z 2022-11-23T02:49:35.3968990Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3969090Z Ran 1 test in 5.825s 2022-11-23T02:49:35.3969096Z 2022-11-23T02:49:35.3969181Z OK 2022-11-23T02:49:35.3969187Z 2022-11-23T02:49:35.3969302Z Generating XML reports... 2022-11-23T02:49:35.3969749Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022251.xml 2022-11-23T02:49:35.3970065Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3970440Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3970662Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3971054Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3971233Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3971239Z 2022-11-23T02:49:35.3971338Z Running tests... 2022-11-23T02:49:35.3971604Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3971920Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60039 2022-11-23T02:49:35.3972125Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60040 2022-11-23T02:49:35.3972377Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3972755Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3972923Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3973307Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3973487Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3973713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3974086Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3974238Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3974622Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3974803Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3975032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3975431Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3975826Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3976038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3976251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3976582Z STAGE:2022-11-23 02:23:04 60040:60040 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3976912Z STAGE:2022-11-23 02:23:04 60039:60039 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3977248Z STAGE:2022-11-23 02:23:04 60039:60039 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3977658Z STAGE:2022-11-23 02:23:04 60039:60039 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3977882Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:49:35.3978220Z STAGE:2022-11-23 02:23:04 60040:60040 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3978569Z STAGE:2022-11-23 02:23:04 60040:60040 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3978788Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:49:35.3979115Z STAGE:2022-11-23 02:23:04 60039:60039 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3979440Z STAGE:2022-11-23 02:23:04 60040:60040 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3979822Z STAGE:2022-11-23 02:23:04 60040:60040 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3980164Z STAGE:2022-11-23 02:23:04 60039:60039 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3980744Z STAGE:2022-11-23 02:23:04 60040:60040 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 02:23:04 60039:60039 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3980751Z 2022-11-23T02:49:35.3981518Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1506: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2022-11-23T02:49:35.3981622Z warnings.warn( 2022-11-23T02:49:35.3982396Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1506: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2022-11-23T02:49:35.3982499Z warnings.warn( 2022-11-23T02:49:35.3982592Z ok (5.317s) 2022-11-23T02:49:35.3982598Z 2022-11-23T02:49:35.3982869Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3982970Z Ran 1 test in 5.318s 2022-11-23T02:49:35.3982977Z 2022-11-23T02:49:35.3983048Z OK 2022-11-23T02:49:35.3983067Z 2022-11-23T02:49:35.3983170Z Generating XML reports... 2022-11-23T02:49:35.3983617Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022301.xml 2022-11-23T02:49:35.3983937Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3984312Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3984475Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3984863Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3985040Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3985046Z 2022-11-23T02:49:35.3985146Z Running tests... 2022-11-23T02:49:35.3985414Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3985717Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60252 2022-11-23T02:49:35.3985923Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60253 2022-11-23T02:49:35.3986181Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3986560Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3986785Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3987173Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3987350Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3987574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3987974Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3988345Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3988509Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3988894Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3989149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3989367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.3989769Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.3989987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.3990321Z STAGE:2022-11-23 02:23:13 60253:60253 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3990533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.3990866Z STAGE:2022-11-23 02:23:13 60252:60252 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3991207Z STAGE:2022-11-23 02:23:13 60252:60252 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3991569Z STAGE:2022-11-23 02:23:13 60252:60252 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3991907Z STAGE:2022-11-23 02:23:13 60253:60253 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3992254Z STAGE:2022-11-23 02:23:13 60253:60253 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3992584Z STAGE:2022-11-23 02:23:13 60252:60252 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3992909Z STAGE:2022-11-23 02:23:13 60253:60253 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.3993243Z STAGE:2022-11-23 02:23:13 60253:60253 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.3993805Z STAGE:2022-11-23 02:23:13 60252:60252 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:23:13 60253:60253 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3993815Z 2022-11-23T02:49:35.3994168Z STAGE:2022-11-23 02:23:13 60252:60252 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.3994262Z ok (5.122s) 2022-11-23T02:49:35.3994268Z 2022-11-23T02:49:35.3994539Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3994639Z Ran 1 test in 5.123s 2022-11-23T02:49:35.3994645Z 2022-11-23T02:49:35.3994727Z OK 2022-11-23T02:49:35.3994733Z 2022-11-23T02:49:35.3994847Z Generating XML reports... 2022-11-23T02:49:35.3995298Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022311.xml 2022-11-23T02:49:35.3995613Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.3995989Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3996155Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3996589Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3996770Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3996790Z 2022-11-23T02:49:35.3996878Z Running tests... 2022-11-23T02:49:35.3997147Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.3997454Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60465 2022-11-23T02:49:35.3997661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60466 2022-11-23T02:49:35.3997915Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.3998287Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3998510Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.3998897Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.3999078Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.3999305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.3999677Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.3999844Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4000231Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4000407Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4000638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4001040Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4001437Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4001652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4001866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4001960Z ok (5.325s) 2022-11-23T02:49:35.4001966Z 2022-11-23T02:49:35.4002234Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4002323Z Ran 1 test in 5.326s 2022-11-23T02:49:35.4002341Z 2022-11-23T02:49:35.4002413Z OK 2022-11-23T02:49:35.4002418Z 2022-11-23T02:49:35.4002533Z Generating XML reports... 2022-11-23T02:49:35.4002982Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022320.xml 2022-11-23T02:49:35.4003300Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4003672Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4003837Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4004219Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4004398Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4004404Z 2022-11-23T02:49:35.4004504Z Running tests... 2022-11-23T02:49:35.4004771Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4005067Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60672 2022-11-23T02:49:35.4005329Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60673 2022-11-23T02:49:35.4005584Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4005961Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4006125Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4006508Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4006689Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4006915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4007357Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4007791Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4007957Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4008331Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4008508Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4008735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4009131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4009343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4009679Z STAGE:2022-11-23 02:23:32 60673:60673 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4009896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4010228Z STAGE:2022-11-23 02:23:33 60672:60672 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4010564Z STAGE:2022-11-23 02:23:33 60672:60672 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4010915Z STAGE:2022-11-23 02:23:33 60672:60672 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4011253Z STAGE:2022-11-23 02:23:33 60673:60673 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4011603Z STAGE:2022-11-23 02:23:33 60673:60673 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4011931Z STAGE:2022-11-23 02:23:33 60672:60672 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4012262Z STAGE:2022-11-23 02:23:33 60673:60673 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4012607Z STAGE:2022-11-23 02:23:33 60672:60672 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4012961Z STAGE:2022-11-23 02:23:33 60672:60672 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4013296Z STAGE:2022-11-23 02:23:33 60673:60673 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4013646Z STAGE:2022-11-23 02:23:33 60673:60673 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4013739Z ok (5.019s) 2022-11-23T02:49:35.4013745Z 2022-11-23T02:49:35.4014014Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4014115Z Ran 1 test in 5.020s 2022-11-23T02:49:35.4014121Z 2022-11-23T02:49:35.4014207Z OK 2022-11-23T02:49:35.4014212Z 2022-11-23T02:49:35.4014327Z Generating XML reports... 2022-11-23T02:49:35.4014763Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022330.xml 2022-11-23T02:49:35.4015147Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4015520Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4015685Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4016073Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4016251Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4016257Z 2022-11-23T02:49:35.4016361Z Running tests... 2022-11-23T02:49:35.4016628Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4016932Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60885 2022-11-23T02:49:35.4017196Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60886 2022-11-23T02:49:35.4017452Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4017838Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4018005Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4018394Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4018579Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4018802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4019204Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4019586Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4019751Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4020135Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4020314Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4020535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4020933Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4021137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4021466Z STAGE:2022-11-23 02:23:42 60886:60886 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4021686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4022017Z STAGE:2022-11-23 02:23:42 60885:60885 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4022353Z STAGE:2022-11-23 02:23:42 60885:60885 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4022685Z STAGE:2022-11-23 02:23:42 60886:60886 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4023048Z STAGE:2022-11-23 02:23:42 60885:60885 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4023401Z STAGE:2022-11-23 02:23:42 60886:60886 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4023735Z STAGE:2022-11-23 02:23:42 60886:60886 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4024065Z STAGE:2022-11-23 02:23:42 60885:60885 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4024469Z STAGE:2022-11-23 02:23:42 60885:60885 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4024810Z STAGE:2022-11-23 02:23:42 60886:60886 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4025168Z STAGE:2022-11-23 02:23:42 60885:60885 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4025522Z STAGE:2022-11-23 02:23:42 60886:60886 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4025623Z ok (4.920s) 2022-11-23T02:49:35.4025629Z 2022-11-23T02:49:35.4025904Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4026011Z Ran 1 test in 4.920s 2022-11-23T02:49:35.4026017Z 2022-11-23T02:49:35.4026107Z OK 2022-11-23T02:49:35.4026112Z 2022-11-23T02:49:35.4026241Z Generating XML reports... 2022-11-23T02:49:35.4026757Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022339.xml 2022-11-23T02:49:35.4027101Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4027490Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4027673Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4028044Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4028238Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4028244Z 2022-11-23T02:49:35.4028363Z Running tests... 2022-11-23T02:49:35.4028650Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4028972Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61098 2022-11-23T02:49:35.4029200Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61099 2022-11-23T02:49:35.4029469Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4029852Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4030033Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4030435Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4030629Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4030867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4031257Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4031441Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4031832Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4032029Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4032266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4032682Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4033093Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4033330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4033672Z STAGE:2022-11-23 02:23:51 61099:61099 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4033963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4034311Z STAGE:2022-11-23 02:23:51 61098:61098 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4034634Z STAGE:2022-11-23 02:23:51 61098:61098 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4034999Z STAGE:2022-11-23 02:23:51 61098:61098 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4035349Z STAGE:2022-11-23 02:23:51 61099:61099 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4035712Z STAGE:2022-11-23 02:23:51 61099:61099 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4036057Z STAGE:2022-11-23 02:23:51 61098:61098 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4036397Z STAGE:2022-11-23 02:23:51 61099:61099 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4036796Z STAGE:2022-11-23 02:23:51 61098:61098 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4037166Z STAGE:2022-11-23 02:23:51 61098:61098 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4037505Z STAGE:2022-11-23 02:23:51 61099:61099 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4037865Z STAGE:2022-11-23 02:23:51 61099:61099 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4037970Z ok (5.024s) 2022-11-23T02:49:35.4037976Z 2022-11-23T02:49:35.4038259Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4038372Z Ran 1 test in 5.024s 2022-11-23T02:49:35.4038378Z 2022-11-23T02:49:35.4038476Z OK 2022-11-23T02:49:35.4038482Z 2022-11-23T02:49:35.4038606Z Generating XML reports... 2022-11-23T02:49:35.4039073Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022348.xml 2022-11-23T02:49:35.4039404Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4039795Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4039978Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4040358Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4040532Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4040538Z 2022-11-23T02:49:35.4040633Z Running tests... 2022-11-23T02:49:35.4040890Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4041190Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61311 2022-11-23T02:49:35.4041401Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61312 2022-11-23T02:49:35.4041651Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4042021Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4042180Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4042557Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4042732Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4042955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4043323Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4043547Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4043940Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4044113Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4044334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4044728Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4045118Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4045327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4045534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4045910Z STAGE:2022-11-23 02:24:01 61312:61312 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4046238Z STAGE:2022-11-23 02:24:01 61311:61311 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4046569Z STAGE:2022-11-23 02:24:01 61311:61311 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4046915Z STAGE:2022-11-23 02:24:01 61311:61311 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4047133Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:49:35.4047466Z STAGE:2022-11-23 02:24:01 61312:61312 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4047974Z STAGE:2022-11-23 02:24:01 61312:61312 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4048197Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:49:35.4048540Z STAGE:2022-11-23 02:24:01 61311:61311 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4048879Z STAGE:2022-11-23 02:24:01 61312:61312 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4049231Z STAGE:2022-11-23 02:24:01 61311:61311 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4049597Z STAGE:2022-11-23 02:24:01 61311:61311 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4049948Z STAGE:2022-11-23 02:24:01 61312:61312 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4050311Z STAGE:2022-11-23 02:24:01 61312:61312 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4050657Z STAGE:2022-11-23 02:24:01 61311:61311 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4050999Z STAGE:2022-11-23 02:24:01 61312:61312 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4051359Z STAGE:2022-11-23 02:24:01 61312:61312 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4051703Z STAGE:2022-11-23 02:24:01 61311:61311 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4052071Z STAGE:2022-11-23 02:24:01 61312:61312 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4052439Z STAGE:2022-11-23 02:24:01 61311:61311 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4052547Z ok (5.725s) 2022-11-23T02:49:35.4052553Z 2022-11-23T02:49:35.4052841Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4052957Z Ran 1 test in 5.726s 2022-11-23T02:49:35.4052963Z 2022-11-23T02:49:35.4053066Z OK 2022-11-23T02:49:35.4053071Z 2022-11-23T02:49:35.4053206Z Generating XML reports... 2022-11-23T02:49:35.4053674Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022358.xml 2022-11-23T02:49:35.4054081Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4054473Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4054652Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4055022Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4055213Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4055219Z 2022-11-23T02:49:35.4055337Z Running tests... 2022-11-23T02:49:35.4055627Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4055954Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61524 2022-11-23T02:49:35.4056240Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61525 2022-11-23T02:49:35.4056521Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4056916Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4057098Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4057499Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4057697Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4057941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4058326Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4058516Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4058921Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4059113Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4059345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4059749Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4060146Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4060362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4060594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4060948Z STAGE:2022-11-23 02:24:11 61525:61525 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4061297Z STAGE:2022-11-23 02:24:11 61524:61524 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4061621Z STAGE:2022-11-23 02:24:11 61524:61524 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4062202Z STAGE:2022-11-23 02:24:11 61525:61525 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:24:11 61524:61524 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4062228Z 2022-11-23T02:49:35.4062438Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:49:35.4062802Z STAGE:2022-11-23 02:24:11 61525:61525 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4063036Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:49:35.4063792Z STAGE:2022-11-23 02:24:11 61524:61524 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 02:24:11 61525:61525 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4063799Z 2022-11-23T02:49:35.4064136Z STAGE:2022-11-23 02:24:11 61524:61524 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4064496Z STAGE:2022-11-23 02:24:11 61524:61524 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4064848Z STAGE:2022-11-23 02:24:11 61525:61525 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4065213Z STAGE:2022-11-23 02:24:11 61525:61525 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4065553Z STAGE:2022-11-23 02:24:11 61524:61524 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4065893Z STAGE:2022-11-23 02:24:11 61525:61525 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4066393Z STAGE:2022-11-23 02:24:11 61525:61525 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4066975Z STAGE:2022-11-23 02:24:11 61524:61524 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:24:11 61525:61525 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4066981Z 2022-11-23T02:49:35.4067348Z STAGE:2022-11-23 02:24:11 61524:61524 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4067453Z ok (6.018s) 2022-11-23T02:49:35.4067460Z 2022-11-23T02:49:35.4067743Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4067857Z Ran 1 test in 6.018s 2022-11-23T02:49:35.4067863Z 2022-11-23T02:49:35.4067961Z OK 2022-11-23T02:49:35.4067967Z 2022-11-23T02:49:35.4068093Z Generating XML reports... 2022-11-23T02:49:35.4068555Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022408.xml 2022-11-23T02:49:35.4068889Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4069276Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4069457Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4069854Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4070048Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4070054Z 2022-11-23T02:49:35.4070169Z Running tests... 2022-11-23T02:49:35.4070457Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4070755Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61737 2022-11-23T02:49:35.4070973Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61738 2022-11-23T02:49:35.4071228Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4071599Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4071758Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4072145Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4072319Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4072538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4072906Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4073124Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4073502Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4073683Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4073919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4074316Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4074706Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4074920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4075139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4075518Z STAGE:2022-11-23 02:24:21 61738:61738 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4075851Z STAGE:2022-11-23 02:24:21 61737:61737 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4076185Z STAGE:2022-11-23 02:24:21 61737:61737 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4076534Z STAGE:2022-11-23 02:24:21 61737:61737 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4076757Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:49:35.4077088Z STAGE:2022-11-23 02:24:21 61738:61738 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4077430Z STAGE:2022-11-23 02:24:21 61738:61738 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4077639Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:49:35.4077974Z STAGE:2022-11-23 02:24:21 61737:61737 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4078299Z STAGE:2022-11-23 02:24:21 61738:61738 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4078635Z STAGE:2022-11-23 02:24:21 61737:61737 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4078981Z STAGE:2022-11-23 02:24:21 61737:61737 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4079316Z STAGE:2022-11-23 02:24:21 61738:61738 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4079659Z STAGE:2022-11-23 02:24:21 61738:61738 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4079986Z STAGE:2022-11-23 02:24:21 61737:61737 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4080310Z STAGE:2022-11-23 02:24:21 61738:61738 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4080644Z STAGE:2022-11-23 02:24:21 61737:61737 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4080994Z STAGE:2022-11-23 02:24:21 61737:61737 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4081327Z STAGE:2022-11-23 02:24:21 61738:61738 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4081670Z STAGE:2022-11-23 02:24:21 61738:61738 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4081759Z ok (5.620s) 2022-11-23T02:49:35.4081765Z 2022-11-23T02:49:35.4082028Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4082127Z Ran 1 test in 5.621s 2022-11-23T02:49:35.4082133Z 2022-11-23T02:49:35.4082218Z OK 2022-11-23T02:49:35.4082223Z 2022-11-23T02:49:35.4082334Z Generating XML reports... 2022-11-23T02:49:35.4082776Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022418.xml 2022-11-23T02:49:35.4083142Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4083524Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4083688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4084066Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4084232Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4084253Z 2022-11-23T02:49:35.4084341Z Running tests... 2022-11-23T02:49:35.4084608Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4084834Z test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s) 2022-11-23T02:49:35.4084840Z 2022-11-23T02:49:35.4085160Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4085260Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4085266Z 2022-11-23T02:49:35.4085362Z OK (skipped=1) 2022-11-23T02:49:35.4085367Z 2022-11-23T02:49:35.4085484Z Generating XML reports... 2022-11-23T02:49:35.4085927Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022428.xml 2022-11-23T02:49:35.4086243Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4086611Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4086771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4087152Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4087336Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4087346Z 2022-11-23T02:49:35.4087446Z Running tests... 2022-11-23T02:49:35.4087767Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4088012Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s) 2022-11-23T02:49:35.4088018Z 2022-11-23T02:49:35.4088287Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4088381Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4088387Z 2022-11-23T02:49:35.4088481Z OK (skipped=1) 2022-11-23T02:49:35.4088487Z 2022-11-23T02:49:35.4088596Z Generating XML reports... 2022-11-23T02:49:35.4089035Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022432.xml 2022-11-23T02:49:35.4089335Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4089711Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4089882Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4090266Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4090444Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4090450Z 2022-11-23T02:49:35.4090548Z Running tests... 2022-11-23T02:49:35.4090814Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4091053Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s) 2022-11-23T02:49:35.4091059Z 2022-11-23T02:49:35.4091322Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4091422Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4091427Z 2022-11-23T02:49:35.4091528Z OK (skipped=1) 2022-11-23T02:49:35.4091600Z 2022-11-23T02:49:35.4091721Z Generating XML reports... 2022-11-23T02:49:35.4092168Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022436.xml 2022-11-23T02:49:35.4092484Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4092855Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4093023Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4093405Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4093582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4093588Z 2022-11-23T02:49:35.4093687Z Running tests... 2022-11-23T02:49:35.4093954Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4094259Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s) 2022-11-23T02:49:35.4094267Z 2022-11-23T02:49:35.4094537Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4094640Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4094646Z 2022-11-23T02:49:35.4094732Z OK (skipped=1) 2022-11-23T02:49:35.4094746Z 2022-11-23T02:49:35.4094849Z Generating XML reports... 2022-11-23T02:49:35.4095301Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022441.xml 2022-11-23T02:49:35.4095618Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4095993Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4096158Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4096541Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4096721Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4096727Z 2022-11-23T02:49:35.4096823Z Running tests... 2022-11-23T02:49:35.4097090Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4097333Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s) 2022-11-23T02:49:35.4097340Z 2022-11-23T02:49:35.4097609Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4097706Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4097712Z 2022-11-23T02:49:35.4097805Z OK (skipped=1) 2022-11-23T02:49:35.4097811Z 2022-11-23T02:49:35.4097927Z Generating XML reports... 2022-11-23T02:49:35.4098367Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022445.xml 2022-11-23T02:49:35.4098689Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4099063Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4099224Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4099607Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4099784Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4099790Z 2022-11-23T02:49:35.4099889Z Running tests... 2022-11-23T02:49:35.4100145Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4100402Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s) 2022-11-23T02:49:35.4100424Z 2022-11-23T02:49:35.4100681Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4100832Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4100838Z 2022-11-23T02:49:35.4100938Z OK (skipped=1) 2022-11-23T02:49:35.4100943Z 2022-11-23T02:49:35.4101058Z Generating XML reports... 2022-11-23T02:49:35.4101511Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022449.xml 2022-11-23T02:49:35.4101829Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4102200Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4102362Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4102749Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4102923Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4102933Z 2022-11-23T02:49:35.4103092Z Running tests... 2022-11-23T02:49:35.4103364Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4103600Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s) 2022-11-23T02:49:35.4103606Z 2022-11-23T02:49:35.4103876Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4103977Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4103982Z 2022-11-23T02:49:35.4104081Z OK (skipped=1) 2022-11-23T02:49:35.4104086Z 2022-11-23T02:49:35.4104197Z Generating XML reports... 2022-11-23T02:49:35.4104634Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022453.xml 2022-11-23T02:49:35.4104942Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4105324Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4105493Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4105863Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4106039Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4106045Z 2022-11-23T02:49:35.4106147Z Running tests... 2022-11-23T02:49:35.4106414Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4106670Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-11-23T02:49:35.4106676Z 2022-11-23T02:49:35.4106942Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4107042Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4107048Z 2022-11-23T02:49:35.4107151Z OK (skipped=1) 2022-11-23T02:49:35.4107158Z 2022-11-23T02:49:35.4107272Z Generating XML reports... 2022-11-23T02:49:35.4107722Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022458.xml 2022-11-23T02:49:35.4108034Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4108408Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4108572Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4108960Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4109136Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4109142Z 2022-11-23T02:49:35.4109243Z Running tests... 2022-11-23T02:49:35.4109509Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4109777Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:49:35.4109841Z 2022-11-23T02:49:35.4110109Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4110205Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4110211Z 2022-11-23T02:49:35.4110311Z OK (skipped=1) 2022-11-23T02:49:35.4110317Z 2022-11-23T02:49:35.4110433Z Generating XML reports... 2022-11-23T02:49:35.4110871Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022502.xml 2022-11-23T02:49:35.4111171Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4111544Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4111702Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4112134Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4112320Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4112326Z 2022-11-23T02:49:35.4112422Z Running tests... 2022-11-23T02:49:35.4112687Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4112963Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2022-11-23T02:49:35.4112970Z 2022-11-23T02:49:35.4113234Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4113328Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4113334Z 2022-11-23T02:49:35.4113425Z OK (skipped=1) 2022-11-23T02:49:35.4113431Z 2022-11-23T02:49:35.4113550Z Generating XML reports... 2022-11-23T02:49:35.4113991Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022506.xml 2022-11-23T02:49:35.4114306Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4114679Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4114843Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4115222Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4115400Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4115405Z 2022-11-23T02:49:35.4115499Z Running tests... 2022-11-23T02:49:35.4115767Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4116043Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-11-23T02:49:35.4116049Z 2022-11-23T02:49:35.4116312Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4116421Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4116427Z 2022-11-23T02:49:35.4116512Z OK (skipped=1) 2022-11-23T02:49:35.4116531Z 2022-11-23T02:49:35.4116633Z Generating XML reports... 2022-11-23T02:49:35.4117073Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022510.xml 2022-11-23T02:49:35.4117390Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4117759Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4117926Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4118314Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4118488Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4118545Z 2022-11-23T02:49:35.4118654Z Running tests... 2022-11-23T02:49:35.4118924Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4119208Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-11-23T02:49:35.4119214Z 2022-11-23T02:49:35.4119482Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4119581Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4119587Z 2022-11-23T02:49:35.4119679Z OK (skipped=1) 2022-11-23T02:49:35.4119684Z 2022-11-23T02:49:35.4119801Z Generating XML reports... 2022-11-23T02:49:35.4120247Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022514.xml 2022-11-23T02:49:35.4120555Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4120983Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4121158Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4121541Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4121722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4121728Z 2022-11-23T02:49:35.4121831Z Running tests... 2022-11-23T02:49:35.4122102Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4122370Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2022-11-23T02:49:35.4122393Z 2022-11-23T02:49:35.4122647Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4122743Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4122748Z 2022-11-23T02:49:35.4122847Z OK (skipped=1) 2022-11-23T02:49:35.4122860Z 2022-11-23T02:49:35.4122979Z Generating XML reports... 2022-11-23T02:49:35.4123417Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022518.xml 2022-11-23T02:49:35.4123739Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4124114Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4124272Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4124666Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4124850Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4124856Z 2022-11-23T02:49:35.4124950Z Running tests... 2022-11-23T02:49:35.4125221Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4125522Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-11-23T02:49:35.4125528Z 2022-11-23T02:49:35.4125791Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4125893Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4125899Z 2022-11-23T02:49:35.4125993Z OK (skipped=1) 2022-11-23T02:49:35.4125999Z 2022-11-23T02:49:35.4126109Z Generating XML reports... 2022-11-23T02:49:35.4126550Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022522.xml 2022-11-23T02:49:35.4126868Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4127248Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4127416Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4127933Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4128099Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4128120Z 2022-11-23T02:49:35.4128207Z Running tests... 2022-11-23T02:49:35.4128480Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4128748Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:49:35.4128754Z 2022-11-23T02:49:35.4129020Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4129125Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4129131Z 2022-11-23T02:49:35.4129225Z OK (skipped=1) 2022-11-23T02:49:35.4129230Z 2022-11-23T02:49:35.4129348Z Generating XML reports... 2022-11-23T02:49:35.4129847Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022526.xml 2022-11-23T02:49:35.4130174Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4130549Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4130717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4131096Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4131282Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4131288Z 2022-11-23T02:49:35.4131390Z Running tests... 2022-11-23T02:49:35.4131652Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4131936Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-11-23T02:49:35.4131950Z 2022-11-23T02:49:35.4132213Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4132318Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4132324Z 2022-11-23T02:49:35.4132428Z OK (skipped=1) 2022-11-23T02:49:35.4132433Z 2022-11-23T02:49:35.4132548Z Generating XML reports... 2022-11-23T02:49:35.4132990Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022530.xml 2022-11-23T02:49:35.4133299Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4133666Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4133817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4134201Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4134390Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4134397Z 2022-11-23T02:49:35.4134491Z Running tests... 2022-11-23T02:49:35.4134763Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4135042Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:49:35.4135048Z 2022-11-23T02:49:35.4135310Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4135413Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4135419Z 2022-11-23T02:49:35.4135513Z OK (skipped=1) 2022-11-23T02:49:35.4135519Z 2022-11-23T02:49:35.4135638Z Generating XML reports... 2022-11-23T02:49:35.4136086Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022535.xml 2022-11-23T02:49:35.4136393Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4136840Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4137009Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4137389Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4137569Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4137575Z 2022-11-23T02:49:35.4137674Z Running tests... 2022-11-23T02:49:35.4137940Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4138224Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2022-11-23T02:49:35.4138230Z 2022-11-23T02:49:35.4138497Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4138599Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4138656Z 2022-11-23T02:49:35.4138755Z OK (skipped=1) 2022-11-23T02:49:35.4138761Z 2022-11-23T02:49:35.4138871Z Generating XML reports... 2022-11-23T02:49:35.4139304Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022539.xml 2022-11-23T02:49:35.4139621Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4139996Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4140163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4140546Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4140719Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4140725Z 2022-11-23T02:49:35.4140828Z Running tests... 2022-11-23T02:49:35.4141106Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4141381Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-11-23T02:49:35.4141388Z 2022-11-23T02:49:35.4141655Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4141762Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4141768Z 2022-11-23T02:49:35.4141864Z OK (skipped=1) 2022-11-23T02:49:35.4141870Z 2022-11-23T02:49:35.4141987Z Generating XML reports... 2022-11-23T02:49:35.4142426Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022543.xml 2022-11-23T02:49:35.4142747Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4143124Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4143298Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4143683Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4143867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4143873Z 2022-11-23T02:49:35.4143975Z Running tests... 2022-11-23T02:49:35.4144246Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4144539Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-11-23T02:49:35.4144545Z 2022-11-23T02:49:35.4144807Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4144895Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4144919Z 2022-11-23T02:49:35.4145004Z OK (skipped=1) 2022-11-23T02:49:35.4145010Z 2022-11-23T02:49:35.4145132Z Generating XML reports... 2022-11-23T02:49:35.4145629Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022547.xml 2022-11-23T02:49:35.4145944Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4146326Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4146493Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4146878Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4147051Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4147057Z 2022-11-23T02:49:35.4147159Z Running tests... 2022-11-23T02:49:35.4147431Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4150171Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:49:35.4150186Z 2022-11-23T02:49:35.4150474Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4150573Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4150578Z 2022-11-23T02:49:35.4150680Z OK (skipped=1) 2022-11-23T02:49:35.4150686Z 2022-11-23T02:49:35.4150805Z Generating XML reports... 2022-11-23T02:49:35.4151244Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022551.xml 2022-11-23T02:49:35.4151562Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4151939Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4152098Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4152482Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4152663Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4152669Z 2022-11-23T02:49:35.4152768Z Running tests... 2022-11-23T02:49:35.4153032Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4153312Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-11-23T02:49:35.4153326Z 2022-11-23T02:49:35.4153578Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4153676Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4153682Z 2022-11-23T02:49:35.4153789Z OK (skipped=1) 2022-11-23T02:49:35.4153794Z 2022-11-23T02:49:35.4153910Z Generating XML reports... 2022-11-23T02:49:35.4154345Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022555.xml 2022-11-23T02:49:35.4154662Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4155035Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4155206Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4155584Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4155759Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4155765Z 2022-11-23T02:49:35.4155863Z Running tests... 2022-11-23T02:49:35.4156130Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4156410Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2022-11-23T02:49:35.4156417Z 2022-11-23T02:49:35.4156681Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4156831Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4156837Z 2022-11-23T02:49:35.4156933Z OK (skipped=1) 2022-11-23T02:49:35.4156938Z 2022-11-23T02:49:35.4157062Z Generating XML reports... 2022-11-23T02:49:35.4157503Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022559.xml 2022-11-23T02:49:35.4157813Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4158182Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4158349Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4158739Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4158916Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4158925Z 2022-11-23T02:49:35.4159060Z Running tests... 2022-11-23T02:49:35.4159330Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4159625Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s) 2022-11-23T02:49:35.4159631Z 2022-11-23T02:49:35.4159902Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4160002Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4160007Z 2022-11-23T02:49:35.4160104Z OK (skipped=1) 2022-11-23T02:49:35.4160110Z 2022-11-23T02:49:35.4160230Z Generating XML reports... 2022-11-23T02:49:35.4160676Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022603.xml 2022-11-23T02:49:35.4160991Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4161365Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4161531Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4161923Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4162110Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4162116Z 2022-11-23T02:49:35.4162215Z Running tests... 2022-11-23T02:49:35.4162479Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4162793Z test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63534 2022-11-23T02:49:35.4163008Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63535 2022-11-23T02:49:35.4163263Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4163647Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4163821Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4164209Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4164386Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4164609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4164967Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4165144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4165535Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4165771Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4165994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4166400Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4166803Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4167020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4167233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4167462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:49:35.4167813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:49:35.4168286Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.4168687Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.4168980Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:49:35.4169256Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:49:35.4169349Z ok (5.764s) 2022-11-23T02:49:35.4169356Z 2022-11-23T02:49:35.4169627Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4169733Z Ran 1 test in 5.764s 2022-11-23T02:49:35.4169739Z 2022-11-23T02:49:35.4169824Z OK 2022-11-23T02:49:35.4169830Z 2022-11-23T02:49:35.4169943Z Generating XML reports... 2022-11-23T02:49:35.4170388Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022608.xml 2022-11-23T02:49:35.4170717Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4171077Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4171243Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4171627Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4171820Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4171826Z 2022-11-23T02:49:35.4171924Z Running tests... 2022-11-23T02:49:35.4172190Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4172628Z test_backend_full_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.001s) 2022-11-23T02:49:35.4172634Z 2022-11-23T02:49:35.4172913Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4173012Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4173017Z 2022-11-23T02:49:35.4173116Z OK (skipped=1) 2022-11-23T02:49:35.4173122Z 2022-11-23T02:49:35.4173244Z Generating XML reports... 2022-11-23T02:49:35.4173692Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022617.xml 2022-11-23T02:49:35.4174002Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4174373Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4174549Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4174931Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4175104Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4175179Z 2022-11-23T02:49:35.4175282Z Running tests... 2022-11-23T02:49:35.4175554Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4175989Z test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.001s) 2022-11-23T02:49:35.4175996Z 2022-11-23T02:49:35.4176256Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4176354Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4176360Z 2022-11-23T02:49:35.4176461Z OK (skipped=1) 2022-11-23T02:49:35.4176467Z 2022-11-23T02:49:35.4176570Z Generating XML reports... 2022-11-23T02:49:35.4177012Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022622.xml 2022-11-23T02:49:35.4177321Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4177744Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4177923Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4178304Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4178481Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4178487Z 2022-11-23T02:49:35.4178584Z Running tests... 2022-11-23T02:49:35.4178855Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4179145Z test_barrier (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63886 2022-11-23T02:49:35.4179346Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63887 2022-11-23T02:49:35.4179599Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4179979Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4180137Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4180513Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4180693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4180920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4181284Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4181454Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4181841Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4182021Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4182250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4182647Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4183024Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4183240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4183461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4183550Z ok (5.660s) 2022-11-23T02:49:35.4183556Z 2022-11-23T02:49:35.4183822Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4183925Z Ran 1 test in 5.661s 2022-11-23T02:49:35.4183986Z 2022-11-23T02:49:35.4184071Z OK 2022-11-23T02:49:35.4184076Z 2022-11-23T02:49:35.4184188Z Generating XML reports... 2022-11-23T02:49:35.4184631Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022626.xml 2022-11-23T02:49:35.4184944Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4185311Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4185481Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4185867Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4186041Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4186047Z 2022-11-23T02:49:35.4186154Z Running tests... 2022-11-23T02:49:35.4186468Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4186767Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64093 2022-11-23T02:49:35.4186968Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64094 2022-11-23T02:49:35.4187227Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4187605Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4187767Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4188155Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4188320Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4188551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4188920Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4189080Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4189463Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4189645Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4189865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4190257Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4190653Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4190874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4191087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4191176Z ok (6.271s) 2022-11-23T02:49:35.4191182Z 2022-11-23T02:49:35.4191445Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4191551Z Ran 1 test in 6.271s 2022-11-23T02:49:35.4191557Z 2022-11-23T02:49:35.4191642Z OK 2022-11-23T02:49:35.4191648Z 2022-11-23T02:49:35.4191757Z Generating XML reports... 2022-11-23T02:49:35.4192204Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022636.xml 2022-11-23T02:49:35.4192520Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4192885Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4193049Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4193491Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4193669Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4193675Z 2022-11-23T02:49:35.4193762Z Running tests... 2022-11-23T02:49:35.4194025Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4194322Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64300 2022-11-23T02:49:35.4194533Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64301 2022-11-23T02:49:35.4194792Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4195219Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4195388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4195774Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4195957Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4196184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4196552Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4196726Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4197112Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4197287Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4197513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4197915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4198316Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4198528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4198737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4198963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:49:35.4199193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:49:35.4199588Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.4199982Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.4200063Z ok (5.823s) 2022-11-23T02:49:35.4200085Z 2022-11-23T02:49:35.4200337Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4200438Z Ran 1 test in 5.824s 2022-11-23T02:49:35.4200444Z 2022-11-23T02:49:35.4200524Z OK 2022-11-23T02:49:35.4200529Z 2022-11-23T02:49:35.4200640Z Generating XML reports... 2022-11-23T02:49:35.4201080Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022646.xml 2022-11-23T02:49:35.4201397Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4201765Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4201928Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4202370Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4202550Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4202556Z 2022-11-23T02:49:35.4202651Z Running tests... 2022-11-23T02:49:35.4202913Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4203225Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64513 2022-11-23T02:49:35.4203435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64514 2022-11-23T02:49:35.4203684Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4204106Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4204282Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4204661Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4204833Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4205053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4205450Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4205820Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4205972Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4206348Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4206528Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4206756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4207155Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4207369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4207579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4207774Z skip: Skipped due to small world size. (5.444s) 2022-11-23T02:49:35.4207781Z 2022-11-23T02:49:35.4208054Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4208149Z Ran 1 test in 5.444s 2022-11-23T02:49:35.4208154Z 2022-11-23T02:49:35.4208247Z OK (skipped=1) 2022-11-23T02:49:35.4208253Z 2022-11-23T02:49:35.4208371Z Generating XML reports... 2022-11-23T02:49:35.4208811Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022656.xml 2022-11-23T02:49:35.4209131Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4209497Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4209659Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4210042Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4210225Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4210231Z 2022-11-23T02:49:35.4210328Z Running tests... 2022-11-23T02:49:35.4210591Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4210890Z test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64720 2022-11-23T02:49:35.4211164Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64721 2022-11-23T02:49:35.4211405Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4211776Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4211937Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4212319Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4212499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4212721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4213141Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4213313Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4213700Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4213873Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4214095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4214485Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4214881Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4215096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4215310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4215463Z skip: Skipped due to small world size. (5.368s) 2022-11-23T02:49:35.4215470Z 2022-11-23T02:49:35.4215740Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4215836Z Ran 1 test in 5.368s 2022-11-23T02:49:35.4215842Z 2022-11-23T02:49:35.4215935Z OK (skipped=1) 2022-11-23T02:49:35.4215941Z 2022-11-23T02:49:35.4216063Z Generating XML reports... 2022-11-23T02:49:35.4216504Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022706.xml 2022-11-23T02:49:35.4216813Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4217180Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4217332Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4217716Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4217893Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4217899Z 2022-11-23T02:49:35.4218004Z Running tests... 2022-11-23T02:49:35.4218269Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4218572Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64927 2022-11-23T02:49:35.4218781Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64928 2022-11-23T02:49:35.4219031Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4219397Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4219618Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4220003Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4220180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4220403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4220793Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4221161Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4221322Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4221699Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4221954Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4222175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4222568Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4222778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4222990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4223135Z skip: Skipped due to small world size. (4.979s) 2022-11-23T02:49:35.4223141Z 2022-11-23T02:49:35.4223394Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4223489Z Ran 1 test in 4.980s 2022-11-23T02:49:35.4223495Z 2022-11-23T02:49:35.4223588Z OK (skipped=1) 2022-11-23T02:49:35.4223593Z 2022-11-23T02:49:35.4223708Z Generating XML reports... 2022-11-23T02:49:35.4224154Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022715.xml 2022-11-23T02:49:35.4224460Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4224827Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4224987Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4225365Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4225540Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4225546Z 2022-11-23T02:49:35.4225641Z Running tests... 2022-11-23T02:49:35.4225903Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4226213Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65134 2022-11-23T02:49:35.4226419Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65135 2022-11-23T02:49:35.4226669Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4227037Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4227196Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4227573Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4227749Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4227973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4228340Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4228557Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4228929Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4229104Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4229323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4229716Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4230102Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4230313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4230573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4230798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:49:35.4231014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:49:35.4231409Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.4231796Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.4231886Z ok (6.875s) 2022-11-23T02:49:35.4231892Z 2022-11-23T02:49:35.4232152Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4232250Z Ran 1 test in 6.875s 2022-11-23T02:49:35.4232255Z 2022-11-23T02:49:35.4232336Z OK 2022-11-23T02:49:35.4232341Z 2022-11-23T02:49:35.4232451Z Generating XML reports... 2022-11-23T02:49:35.4232896Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022724.xml 2022-11-23T02:49:35.4233207Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4233574Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4233736Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4234117Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4234292Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4234298Z 2022-11-23T02:49:35.4234394Z Running tests... 2022-11-23T02:49:35.4234647Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4234952Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65347 2022-11-23T02:49:35.4235156Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65348 2022-11-23T02:49:35.4235406Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4235774Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4235936Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4236313Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4236487Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4236707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4237078Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4237296Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4237677Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4237851Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4238070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4238462Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4238852Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4239064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4239319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4239543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4239759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4240152Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4240538Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4240628Z ok (6.264s) 2022-11-23T02:49:35.4240634Z 2022-11-23T02:49:35.4240896Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4240985Z Ran 1 test in 6.264s 2022-11-23T02:49:35.4240990Z 2022-11-23T02:49:35.4241071Z OK 2022-11-23T02:49:35.4241076Z 2022-11-23T02:49:35.4241189Z Generating XML reports... 2022-11-23T02:49:35.4241635Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022735.xml 2022-11-23T02:49:35.4241942Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4242309Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4242470Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4242849Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4243025Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4243031Z 2022-11-23T02:49:35.4243127Z Running tests... 2022-11-23T02:49:35.4243388Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4243692Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65560 2022-11-23T02:49:35.4243896Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65561 2022-11-23T02:49:35.4244145Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4244515Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4244675Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4245055Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4245228Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4245447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4245842Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4246262Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4246422Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4246792Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4246964Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4247183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4247575Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4247840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4248106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4248259Z skip: Skipped due to small world size. (5.161s) 2022-11-23T02:49:35.4248265Z 2022-11-23T02:49:35.4248535Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4248632Z Ran 1 test in 5.161s 2022-11-23T02:49:35.4248638Z 2022-11-23T02:49:35.4248730Z OK (skipped=1) 2022-11-23T02:49:35.4248735Z 2022-11-23T02:49:35.4248847Z Generating XML reports... 2022-11-23T02:49:35.4249284Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022746.xml 2022-11-23T02:49:35.4249597Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4249965Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4250126Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4250511Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4250685Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4250691Z 2022-11-23T02:49:35.4250787Z Running tests... 2022-11-23T02:49:35.4251053Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4251357Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65767 2022-11-23T02:49:35.4251559Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65768 2022-11-23T02:49:35.4251808Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4252175Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4252334Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4252710Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4252883Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4253104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4253468Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4253627Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4254005Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4254179Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4254401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4254859Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4255246Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4255454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4255662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4255754Z ok (5.063s) 2022-11-23T02:49:35.4255761Z 2022-11-23T02:49:35.4256023Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4256117Z Ran 1 test in 5.063s 2022-11-23T02:49:35.4256123Z 2022-11-23T02:49:35.4256203Z OK 2022-11-23T02:49:35.4256208Z 2022-11-23T02:49:35.4256320Z Generating XML reports... 2022-11-23T02:49:35.4256801Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022755.xml 2022-11-23T02:49:35.4257117Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4257486Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4257649Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4258016Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4258190Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4258196Z 2022-11-23T02:49:35.4258293Z Running tests... 2022-11-23T02:49:35.4258555Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4258863Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65974 2022-11-23T02:49:35.4259074Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65975 2022-11-23T02:49:35.4259328Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4259696Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4259856Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4260233Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4260408Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4260630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4260997Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4261167Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4261546Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4261722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4261946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4262336Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4262724Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4262934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4263145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4263305Z ok (5.073s) 2022-11-23T02:49:35.4263312Z 2022-11-23T02:49:35.4263577Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4263666Z Ran 1 test in 5.073s 2022-11-23T02:49:35.4263671Z 2022-11-23T02:49:35.4263752Z OK 2022-11-23T02:49:35.4263757Z 2022-11-23T02:49:35.4263870Z Generating XML reports... 2022-11-23T02:49:35.4264307Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022804.xml 2022-11-23T02:49:35.4264616Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4264982Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4265144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4265521Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4265771Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4265779Z 2022-11-23T02:49:35.4265879Z Running tests... 2022-11-23T02:49:35.4266145Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4266402Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-11-23T02:49:35.4266408Z 2022-11-23T02:49:35.4266668Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4266766Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4266772Z 2022-11-23T02:49:35.4266866Z OK (skipped=1) 2022-11-23T02:49:35.4266872Z 2022-11-23T02:49:35.4266983Z Generating XML reports... 2022-11-23T02:49:35.4267422Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022813.xml 2022-11-23T02:49:35.4267735Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4268105Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4268266Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4268642Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4268819Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4268825Z 2022-11-23T02:49:35.4268912Z Running tests... 2022-11-23T02:49:35.4269176Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4269413Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s) 2022-11-23T02:49:35.4269420Z 2022-11-23T02:49:35.4269677Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4269774Z Ran 1 test in 0.003s 2022-11-23T02:49:35.4269783Z 2022-11-23T02:49:35.4269881Z OK (skipped=1) 2022-11-23T02:49:35.4269887Z 2022-11-23T02:49:35.4270001Z Generating XML reports... 2022-11-23T02:49:35.4270439Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022818.xml 2022-11-23T02:49:35.4270749Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4271120Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4271281Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4271660Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4271836Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4271842Z 2022-11-23T02:49:35.4271940Z Running tests... 2022-11-23T02:49:35.4272276Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4272524Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-11-23T02:49:35.4272530Z 2022-11-23T02:49:35.4272788Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4272885Z Ran 1 test in 0.003s 2022-11-23T02:49:35.4272891Z 2022-11-23T02:49:35.4272984Z OK (skipped=1) 2022-11-23T02:49:35.4272990Z 2022-11-23T02:49:35.4273101Z Generating XML reports... 2022-11-23T02:49:35.4273534Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022822.xml 2022-11-23T02:49:35.4273845Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4274213Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4274412Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4274800Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4274973Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4274978Z 2022-11-23T02:49:35.4275075Z Running tests... 2022-11-23T02:49:35.4275340Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4275579Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-11-23T02:49:35.4275586Z 2022-11-23T02:49:35.4275846Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4275943Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4275949Z 2022-11-23T02:49:35.4276041Z OK (skipped=1) 2022-11-23T02:49:35.4276047Z 2022-11-23T02:49:35.4276159Z Generating XML reports... 2022-11-23T02:49:35.4276599Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022826.xml 2022-11-23T02:49:35.4276913Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4277282Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4277441Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4277819Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4277997Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4278002Z 2022-11-23T02:49:35.4278098Z Running tests... 2022-11-23T02:49:35.4278361Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4278602Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-11-23T02:49:35.4278610Z 2022-11-23T02:49:35.4278871Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4278968Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4278973Z 2022-11-23T02:49:35.4279068Z OK (skipped=1) 2022-11-23T02:49:35.4279073Z 2022-11-23T02:49:35.4279176Z Generating XML reports... 2022-11-23T02:49:35.4279607Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022830.xml 2022-11-23T02:49:35.4279917Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4280287Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4280450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4280828Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4281059Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4281065Z 2022-11-23T02:49:35.4281164Z Running tests... 2022-11-23T02:49:35.4281431Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4281689Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-11-23T02:49:35.4281695Z 2022-11-23T02:49:35.4281957Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4282051Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4282057Z 2022-11-23T02:49:35.4282153Z OK (skipped=1) 2022-11-23T02:49:35.4282158Z 2022-11-23T02:49:35.4282268Z Generating XML reports... 2022-11-23T02:49:35.4282704Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022834.xml 2022-11-23T02:49:35.4283012Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4283433Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4283598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4283980Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4284153Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4284159Z 2022-11-23T02:49:35.4284256Z Running tests... 2022-11-23T02:49:35.4284517Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4284749Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-11-23T02:49:35.4284763Z 2022-11-23T02:49:35.4285014Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4285114Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4285119Z 2022-11-23T02:49:35.4285221Z OK (skipped=1) 2022-11-23T02:49:35.4285227Z 2022-11-23T02:49:35.4285337Z Generating XML reports... 2022-11-23T02:49:35.4285774Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022838.xml 2022-11-23T02:49:35.4286084Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4286453Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4286613Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4286992Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4287165Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4287171Z 2022-11-23T02:49:35.4287267Z Running tests... 2022-11-23T02:49:35.4287533Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4287888Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s) 2022-11-23T02:49:35.4287895Z 2022-11-23T02:49:35.4288159Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4288256Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4288261Z 2022-11-23T02:49:35.4288354Z OK (skipped=1) 2022-11-23T02:49:35.4288360Z 2022-11-23T02:49:35.4288471Z Generating XML reports... 2022-11-23T02:49:35.4288908Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022842.xml 2022-11-23T02:49:35.4289215Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4289584Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4289745Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4290204Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4290370Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4290386Z 2022-11-23T02:49:35.4290473Z Running tests... 2022-11-23T02:49:35.4290735Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4291023Z test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66709 2022-11-23T02:49:35.4291226Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66710 2022-11-23T02:49:35.4291479Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4291846Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4292061Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4292443Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4292616Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4292837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4293207Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4293368Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4293745Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4293918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4294142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4294535Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4294924Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4295136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4295346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4295676Z STAGE:2022-11-23 02:28:49 66709:66709 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4296000Z STAGE:2022-11-23 02:28:49 66710:66710 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4296338Z STAGE:2022-11-23 02:28:49 66710:66710 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4296672Z STAGE:2022-11-23 02:28:49 66709:66709 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4297014Z STAGE:2022-11-23 02:28:49 66710:66710 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4297360Z STAGE:2022-11-23 02:28:49 66709:66709 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4297685Z STAGE:2022-11-23 02:28:49 66709:66709 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4298010Z STAGE:2022-11-23 02:28:49 66710:66710 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4298343Z STAGE:2022-11-23 02:28:49 66710:66710 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4298691Z STAGE:2022-11-23 02:28:49 66710:66710 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4299024Z STAGE:2022-11-23 02:28:49 66709:66709 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4299433Z STAGE:2022-11-23 02:28:49 66709:66709 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4299762Z STAGE:2022-11-23 02:28:49 66710:66710 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4300087Z STAGE:2022-11-23 02:28:49 66709:66709 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4300421Z STAGE:2022-11-23 02:28:49 66709:66709 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4300765Z STAGE:2022-11-23 02:28:49 66709:66709 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4301095Z STAGE:2022-11-23 02:28:49 66710:66710 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4301441Z STAGE:2022-11-23 02:28:49 66710:66710 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4301767Z STAGE:2022-11-23 02:28:49 66709:66709 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4302139Z STAGE:2022-11-23 02:28:49 66710:66710 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4302476Z STAGE:2022-11-23 02:28:49 66710:66710 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4302821Z STAGE:2022-11-23 02:28:49 66710:66710 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4303150Z STAGE:2022-11-23 02:28:49 66709:66709 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4303493Z STAGE:2022-11-23 02:28:49 66709:66709 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4303818Z STAGE:2022-11-23 02:28:49 66710:66710 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4304138Z STAGE:2022-11-23 02:28:49 66709:66709 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4304472Z STAGE:2022-11-23 02:28:49 66709:66709 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4305042Z STAGE:2022-11-23 02:28:49 66709:66709 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 02:28:49 66710:66710 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4305049Z 2022-11-23T02:49:35.4305397Z STAGE:2022-11-23 02:28:49 66710:66710 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4305724Z STAGE:2022-11-23 02:28:49 66709:66709 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4306035Z STAGE:2022-11-23 02:28:49 66710:66710 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4306366Z STAGE:2022-11-23 02:28:49 66710:66710 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4306711Z STAGE:2022-11-23 02:28:49 66710:66710 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4307045Z STAGE:2022-11-23 02:28:49 66709:66709 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4307392Z STAGE:2022-11-23 02:28:49 66709:66709 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4307714Z STAGE:2022-11-23 02:28:49 66710:66710 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4308037Z STAGE:2022-11-23 02:28:49 66709:66709 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4308368Z STAGE:2022-11-23 02:28:49 66709:66709 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4308712Z STAGE:2022-11-23 02:28:49 66709:66709 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4309042Z STAGE:2022-11-23 02:28:49 66710:66710 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4309386Z STAGE:2022-11-23 02:28:49 66710:66710 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4309775Z STAGE:2022-11-23 02:28:49 66709:66709 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4310096Z STAGE:2022-11-23 02:28:49 66710:66710 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4310425Z STAGE:2022-11-23 02:28:49 66710:66710 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4310770Z STAGE:2022-11-23 02:28:49 66710:66710 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4311102Z STAGE:2022-11-23 02:28:49 66709:66709 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4311446Z STAGE:2022-11-23 02:28:49 66709:66709 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4311769Z STAGE:2022-11-23 02:28:49 66710:66710 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4312091Z STAGE:2022-11-23 02:28:49 66709:66709 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4312471Z STAGE:2022-11-23 02:28:49 66709:66709 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4312805Z STAGE:2022-11-23 02:28:49 66710:66710 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4313151Z STAGE:2022-11-23 02:28:49 66709:66709 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4313495Z STAGE:2022-11-23 02:28:49 66710:66710 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4313820Z STAGE:2022-11-23 02:28:49 66709:66709 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4314134Z STAGE:2022-11-23 02:28:49 66710:66710 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4314466Z STAGE:2022-11-23 02:28:49 66710:66710 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4314816Z STAGE:2022-11-23 02:28:49 66710:66710 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4315147Z STAGE:2022-11-23 02:28:49 66709:66709 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4315493Z STAGE:2022-11-23 02:28:49 66709:66709 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4315815Z STAGE:2022-11-23 02:28:49 66710:66710 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4316137Z STAGE:2022-11-23 02:28:49 66709:66709 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4316470Z STAGE:2022-11-23 02:28:49 66709:66709 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4316817Z STAGE:2022-11-23 02:28:49 66709:66709 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4317146Z STAGE:2022-11-23 02:28:49 66710:66710 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4317494Z STAGE:2022-11-23 02:28:49 66710:66710 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4317822Z STAGE:2022-11-23 02:28:49 66709:66709 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4318144Z STAGE:2022-11-23 02:28:49 66710:66710 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4318475Z STAGE:2022-11-23 02:28:49 66710:66710 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4318818Z STAGE:2022-11-23 02:28:49 66710:66710 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4319147Z STAGE:2022-11-23 02:28:49 66709:66709 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4319492Z STAGE:2022-11-23 02:28:49 66709:66709 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4319583Z ok (5.085s) 2022-11-23T02:49:35.4319589Z 2022-11-23T02:49:35.4319853Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4320007Z Ran 1 test in 5.085s 2022-11-23T02:49:35.4320013Z 2022-11-23T02:49:35.4320094Z OK 2022-11-23T02:49:35.4320099Z 2022-11-23T02:49:35.4320212Z Generating XML reports... 2022-11-23T02:49:35.4320654Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022846.xml 2022-11-23T02:49:35.4320956Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4321326Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4321488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4321867Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4322039Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4322045Z 2022-11-23T02:49:35.4322145Z Running tests... 2022-11-23T02:49:35.4322456Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4323357Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.639s) 2022-11-23T02:49:35.4323364Z 2022-11-23T02:49:35.4323623Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4323721Z Ran 1 test in 0.639s 2022-11-23T02:49:35.4323727Z 2022-11-23T02:49:35.4323822Z OK (skipped=1) 2022-11-23T02:49:35.4323827Z 2022-11-23T02:49:35.4323939Z Generating XML reports... 2022-11-23T02:49:35.4324376Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022856.xml 2022-11-23T02:49:35.4324694Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4325064Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4325225Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4325603Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4325779Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4325785Z 2022-11-23T02:49:35.4325881Z Running tests... 2022-11-23T02:49:35.4326142Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4326445Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66988 2022-11-23T02:49:35.4326653Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66989 2022-11-23T02:49:35.4326914Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4327283Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4327444Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4327880Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4328045Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4328265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4328631Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4328794Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4329240Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4329416Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4329635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4330030Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4330418Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4330631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4330843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4331123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:49:35.4331350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:49:35.4331747Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.4332132Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.4332461Z STAGE:2022-11-23 02:29:03 66988:66988 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4332786Z STAGE:2022-11-23 02:29:03 66989:66989 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4333119Z STAGE:2022-11-23 02:29:03 66988:66988 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4333464Z STAGE:2022-11-23 02:29:03 66988:66988 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4333802Z STAGE:2022-11-23 02:29:03 66989:66989 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4334149Z STAGE:2022-11-23 02:29:03 66989:66989 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4334477Z STAGE:2022-11-23 02:29:03 66988:66988 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4334799Z STAGE:2022-11-23 02:29:03 66989:66989 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4335131Z STAGE:2022-11-23 02:29:03 66989:66989 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4335468Z STAGE:2022-11-23 02:29:03 66989:66989 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4335797Z STAGE:2022-11-23 02:29:03 66988:66988 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4336141Z STAGE:2022-11-23 02:29:03 66988:66988 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4336467Z STAGE:2022-11-23 02:29:03 66989:66989 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4336790Z STAGE:2022-11-23 02:29:03 66988:66988 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4337119Z STAGE:2022-11-23 02:29:03 66988:66988 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4337464Z STAGE:2022-11-23 02:29:03 66988:66988 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4337795Z STAGE:2022-11-23 02:29:03 66989:66989 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4338142Z STAGE:2022-11-23 02:29:03 66989:66989 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4338465Z STAGE:2022-11-23 02:29:03 66988:66988 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4338789Z STAGE:2022-11-23 02:29:03 66989:66989 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4339178Z STAGE:2022-11-23 02:29:03 66989:66989 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4339738Z STAGE:2022-11-23 02:29:03 66988:66988 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:29:03 66989:66989 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4339745Z 2022-11-23T02:49:35.4340092Z STAGE:2022-11-23 02:29:03 66988:66988 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4340416Z STAGE:2022-11-23 02:29:03 66989:66989 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4340743Z STAGE:2022-11-23 02:29:03 66988:66988 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4341078Z STAGE:2022-11-23 02:29:03 66988:66988 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4341466Z STAGE:2022-11-23 02:29:03 66988:66988 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4341805Z STAGE:2022-11-23 02:29:03 66989:66989 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4342152Z STAGE:2022-11-23 02:29:03 66989:66989 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4342479Z STAGE:2022-11-23 02:29:03 66988:66988 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4342800Z STAGE:2022-11-23 02:29:03 66989:66989 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4343133Z STAGE:2022-11-23 02:29:03 66989:66989 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4343460Z STAGE:2022-11-23 02:29:03 66988:66988 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4343803Z STAGE:2022-11-23 02:29:03 66989:66989 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4344153Z STAGE:2022-11-23 02:29:03 66988:66988 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4344689Z STAGE:2022-11-23 02:29:03 66989:66989 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 02:29:03 66988:66988 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4344696Z 2022-11-23T02:49:35.4345028Z STAGE:2022-11-23 02:29:03 66988:66988 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4345373Z STAGE:2022-11-23 02:29:03 66988:66988 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4345694Z STAGE:2022-11-23 02:29:03 66989:66989 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4346041Z STAGE:2022-11-23 02:29:03 66989:66989 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4346364Z STAGE:2022-11-23 02:29:03 66988:66988 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4346692Z STAGE:2022-11-23 02:29:03 66989:66989 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4347023Z STAGE:2022-11-23 02:29:03 66988:66988 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4347371Z STAGE:2022-11-23 02:29:03 66988:66988 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4347702Z STAGE:2022-11-23 02:29:03 66989:66989 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4348045Z STAGE:2022-11-23 02:29:03 66989:66989 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4348373Z STAGE:2022-11-23 02:29:03 66988:66988 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4348696Z STAGE:2022-11-23 02:29:03 66989:66989 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4349028Z STAGE:2022-11-23 02:29:03 66988:66988 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4349431Z STAGE:2022-11-23 02:29:03 66988:66988 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4349760Z STAGE:2022-11-23 02:29:03 66989:66989 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4350105Z STAGE:2022-11-23 02:29:03 66989:66989 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4350428Z STAGE:2022-11-23 02:29:03 66988:66988 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4350755Z STAGE:2022-11-23 02:29:03 66989:66989 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4351085Z STAGE:2022-11-23 02:29:03 66989:66989 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4351428Z STAGE:2022-11-23 02:29:03 66989:66989 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4351807Z STAGE:2022-11-23 02:29:03 66988:66988 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4352160Z STAGE:2022-11-23 02:29:03 66988:66988 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4352484Z STAGE:2022-11-23 02:29:03 66989:66989 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4352803Z STAGE:2022-11-23 02:29:03 66988:66988 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4353135Z STAGE:2022-11-23 02:29:03 66988:66988 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4353697Z STAGE:2022-11-23 02:29:03 66989:66989 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:29:03 66988:66988 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4353704Z 2022-11-23T02:49:35.4354048Z STAGE:2022-11-23 02:29:03 66989:66989 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4354374Z STAGE:2022-11-23 02:29:03 66988:66988 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4354687Z STAGE:2022-11-23 02:29:03 66989:66989 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4355018Z STAGE:2022-11-23 02:29:03 66989:66989 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4355572Z STAGE:2022-11-23 02:29:03 66989:66989 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 02:29:03 66988:66988 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4355588Z 2022-11-23T02:49:35.4355924Z STAGE:2022-11-23 02:29:03 66988:66988 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4356015Z ok (5.145s) 2022-11-23T02:49:35.4356023Z 2022-11-23T02:49:35.4356285Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4356381Z Ran 1 test in 5.146s 2022-11-23T02:49:35.4356387Z 2022-11-23T02:49:35.4356471Z OK 2022-11-23T02:49:35.4356480Z 2022-11-23T02:49:35.4356593Z Generating XML reports... 2022-11-23T02:49:35.4357032Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022900.xml 2022-11-23T02:49:35.4357342Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4357710Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4357872Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4358252Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4358426Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4358432Z 2022-11-23T02:49:35.4358527Z Running tests... 2022-11-23T02:49:35.4358790Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4359215Z test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67207 2022-11-23T02:49:35.4359421Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67208 2022-11-23T02:49:35.4359670Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4360043Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4360206Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4360585Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4360760Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4360982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4361388Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4361557Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4361937Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4362113Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4362335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4362728Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4363118Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4363336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4363552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4363696Z skip: Skipped due to small world size. (5.360s) 2022-11-23T02:49:35.4363703Z 2022-11-23T02:49:35.4363965Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4364062Z Ran 1 test in 5.361s 2022-11-23T02:49:35.4364068Z 2022-11-23T02:49:35.4364161Z OK (skipped=1) 2022-11-23T02:49:35.4364167Z 2022-11-23T02:49:35.4364279Z Generating XML reports... 2022-11-23T02:49:35.4364718Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022910.xml 2022-11-23T02:49:35.4365028Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4365395Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4365563Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4365938Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4366113Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4366119Z 2022-11-23T02:49:35.4366216Z Running tests... 2022-11-23T02:49:35.4366482Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4366771Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67414 2022-11-23T02:49:35.4366975Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67415 2022-11-23T02:49:35.4367224Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4367595Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4367849Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4368232Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4368408Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4368631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4368997Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4369157Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4369533Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4369707Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4369988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4370386Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4370775Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4370987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4371197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4371974Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1402: UserWarning: torch.distributed.broadcast_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2022-11-23T02:49:35.4372074Z warnings.warn( 2022-11-23T02:49:35.4372844Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1402: UserWarning: torch.distributed.broadcast_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2022-11-23T02:49:35.4372942Z warnings.warn( 2022-11-23T02:49:35.4373032Z ok (5.463s) 2022-11-23T02:49:35.4373038Z 2022-11-23T02:49:35.4373300Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4373396Z Ran 1 test in 5.463s 2022-11-23T02:49:35.4373402Z 2022-11-23T02:49:35.4373484Z OK 2022-11-23T02:49:35.4373490Z 2022-11-23T02:49:35.4373592Z Generating XML reports... 2022-11-23T02:49:35.4374032Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022919.xml 2022-11-23T02:49:35.4374342Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4374716Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4374878Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4375259Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4375434Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4375440Z 2022-11-23T02:49:35.4375538Z Running tests... 2022-11-23T02:49:35.4375799Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4376702Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.640s) 2022-11-23T02:49:35.4376771Z 2022-11-23T02:49:35.4377038Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4377134Z Ran 1 test in 0.640s 2022-11-23T02:49:35.4377140Z 2022-11-23T02:49:35.4377234Z OK (skipped=1) 2022-11-23T02:49:35.4377239Z 2022-11-23T02:49:35.4377350Z Generating XML reports... 2022-11-23T02:49:35.4377787Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022929.xml 2022-11-23T02:49:35.4378096Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4378462Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4378624Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4379001Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4379232Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4379239Z 2022-11-23T02:49:35.4379338Z Running tests... 2022-11-23T02:49:35.4379603Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4380125Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.001s) 2022-11-23T02:49:35.4380132Z 2022-11-23T02:49:35.4380391Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4380492Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4380498Z 2022-11-23T02:49:35.4380591Z OK (skipped=1) 2022-11-23T02:49:35.4380596Z 2022-11-23T02:49:35.4380708Z Generating XML reports... 2022-11-23T02:49:35.4381132Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022934.xml 2022-11-23T02:49:35.4381451Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4381820Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4381981Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4382361Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4382535Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4382541Z 2022-11-23T02:49:35.4382640Z Running tests... 2022-11-23T02:49:35.4382908Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4383421Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.001s) 2022-11-23T02:49:35.4383427Z 2022-11-23T02:49:35.4383694Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4383791Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4383797Z 2022-11-23T02:49:35.4383891Z OK (skipped=1) 2022-11-23T02:49:35.4383897Z 2022-11-23T02:49:35.4384007Z Generating XML reports... 2022-11-23T02:49:35.4384442Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022938.xml 2022-11-23T02:49:35.4384754Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4385121Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4385283Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4385659Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4385837Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4385895Z 2022-11-23T02:49:35.4385993Z Running tests... 2022-11-23T02:49:35.4386258Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4386561Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67819 2022-11-23T02:49:35.4386769Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67820 2022-11-23T02:49:35.4387013Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4387382Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4387544Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4387921Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4388149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4388372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4388740Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4388900Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4389277Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4389452Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4389677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4390072Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4390473Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4390686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4390896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4391129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8c3kmb9c 2022-11-23T02:49:35.4391377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8c3kmb9c/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4391607Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp31vc3s9i 2022-11-23T02:49:35.4391850Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp31vc3s9i/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4392069Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4392291Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4392381Z ok (7.774s) 2022-11-23T02:49:35.4392387Z 2022-11-23T02:49:35.4392644Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4392743Z Ran 1 test in 7.774s 2022-11-23T02:49:35.4392749Z 2022-11-23T02:49:35.4392831Z OK 2022-11-23T02:49:35.4392836Z 2022-11-23T02:49:35.4392948Z Generating XML reports... 2022-11-23T02:49:35.4393384Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022942.xml 2022-11-23T02:49:35.4393695Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4394062Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4394226Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4394607Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4394839Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4394846Z 2022-11-23T02:49:35.4394942Z Running tests... 2022-11-23T02:49:35.4395204Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4395515Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68036 2022-11-23T02:49:35.4395717Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68037 2022-11-23T02:49:35.4395969Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4396338Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4396542Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4396928Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4397100Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4397325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4397714Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4398078Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4398238Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4398605Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4398787Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4399013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4399401Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4399612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4399826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4400057Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvdbtg9jq 2022-11-23T02:49:35.4400304Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvdbtg9jq/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4400536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbc3o228j 2022-11-23T02:49:35.4400784Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbc3o228j/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4401005Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4401219Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4401430Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4401644Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4401733Z ok (7.373s) 2022-11-23T02:49:35.4401739Z 2022-11-23T02:49:35.4402002Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4402097Z Ran 1 test in 7.373s 2022-11-23T02:49:35.4402103Z 2022-11-23T02:49:35.4402187Z OK 2022-11-23T02:49:35.4402193Z 2022-11-23T02:49:35.4402305Z Generating XML reports... 2022-11-23T02:49:35.4402744Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022954.xml 2022-11-23T02:49:35.4403127Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4403486Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4403648Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4404029Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4404205Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4404211Z 2022-11-23T02:49:35.4404309Z Running tests... 2022-11-23T02:49:35.4404574Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4405528Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78641 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.637s) 2022-11-23T02:49:35.4405541Z 2022-11-23T02:49:35.4405805Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4405903Z Ran 1 test in 0.637s 2022-11-23T02:49:35.4405908Z 2022-11-23T02:49:35.4406005Z OK (skipped=1) 2022-11-23T02:49:35.4406011Z 2022-11-23T02:49:35.4406121Z Generating XML reports... 2022-11-23T02:49:35.4406559Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023005.xml 2022-11-23T02:49:35.4406870Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4407242Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4407402Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4407899Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4408076Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4408081Z 2022-11-23T02:49:35.4408176Z Running tests... 2022-11-23T02:49:35.4408442Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4409377Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.633s) 2022-11-23T02:49:35.4409384Z 2022-11-23T02:49:35.4409646Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4409742Z Ran 1 test in 0.633s 2022-11-23T02:49:35.4409754Z 2022-11-23T02:49:35.4409847Z OK (skipped=1) 2022-11-23T02:49:35.4409852Z 2022-11-23T02:49:35.4409963Z Generating XML reports... 2022-11-23T02:49:35.4410398Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023010.xml 2022-11-23T02:49:35.4410707Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4411076Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4411237Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4411616Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4411781Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4411795Z 2022-11-23T02:49:35.4411882Z Running tests... 2022-11-23T02:49:35.4412221Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4412685Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.003s) 2022-11-23T02:49:35.4412693Z 2022-11-23T02:49:35.4412952Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4413049Z Ran 1 test in 0.004s 2022-11-23T02:49:35.4413055Z 2022-11-23T02:49:35.4413148Z OK (skipped=1) 2022-11-23T02:49:35.4413154Z 2022-11-23T02:49:35.4413268Z Generating XML reports... 2022-11-23T02:49:35.4413704Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023015.xml 2022-11-23T02:49:35.4414015Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4414384Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4414604Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4414992Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4415169Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4415175Z 2022-11-23T02:49:35.4415271Z Running tests... 2022-11-23T02:49:35.4415534Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4415880Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68451 2022-11-23T02:49:35.4416084Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68452 2022-11-23T02:49:35.4416337Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4416716Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4416878Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4417257Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4417435Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4417648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4418036Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4418404Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4418565Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4418950Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4419127Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4419350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4419742Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4419951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4420165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4420399Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzh13337p 2022-11-23T02:49:35.4420640Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzh13337p/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4420875Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5og9pgai 2022-11-23T02:49:35.4421175Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5og9pgai/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4421264Z ok (5.269s) 2022-11-23T02:49:35.4421270Z 2022-11-23T02:49:35.4421541Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4421640Z Ran 1 test in 5.269s 2022-11-23T02:49:35.4421645Z 2022-11-23T02:49:35.4421728Z OK 2022-11-23T02:49:35.4421733Z 2022-11-23T02:49:35.4421846Z Generating XML reports... 2022-11-23T02:49:35.4422283Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023019.xml 2022-11-23T02:49:35.4422589Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4422956Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4423160Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4423546Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4423725Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4423731Z 2022-11-23T02:49:35.4423828Z Running tests... 2022-11-23T02:49:35.4424088Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4424390Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68658 2022-11-23T02:49:35.4424594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68659 2022-11-23T02:49:35.4424847Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4425220Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4425386Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4425767Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4425941Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4426165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4426530Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4426691Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4427070Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4427245Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4427472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4427862Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4428252Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4428469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4428680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4428910Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbyu4530c 2022-11-23T02:49:35.4429144Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbyu4530c/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4429381Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc8xsba8n 2022-11-23T02:49:35.4429685Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc8xsba8n/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4429900Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4430115Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4430206Z ok (7.569s) 2022-11-23T02:49:35.4430212Z 2022-11-23T02:49:35.4430480Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4430578Z Ran 1 test in 7.570s 2022-11-23T02:49:35.4430584Z 2022-11-23T02:49:35.4430664Z OK 2022-11-23T02:49:35.4430670Z 2022-11-23T02:49:35.4430781Z Generating XML reports... 2022-11-23T02:49:35.4431217Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023028.xml 2022-11-23T02:49:35.4431527Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4431950Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4432112Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4432496Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4432677Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4432683Z 2022-11-23T02:49:35.4432779Z Running tests... 2022-11-23T02:49:35.4433042Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4433505Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.005s) 2022-11-23T02:49:35.4433511Z 2022-11-23T02:49:35.4433772Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4433870Z Ran 1 test in 0.005s 2022-11-23T02:49:35.4433883Z 2022-11-23T02:49:35.4433976Z OK (skipped=1) 2022-11-23T02:49:35.4433981Z 2022-11-23T02:49:35.4434092Z Generating XML reports... 2022-11-23T02:49:35.4434517Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023040.xml 2022-11-23T02:49:35.4434826Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4435194Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4435355Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4435731Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4435907Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4435912Z 2022-11-23T02:49:35.4436010Z Running tests... 2022-11-23T02:49:35.4436279Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4436732Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.004s) 2022-11-23T02:49:35.4436740Z 2022-11-23T02:49:35.4436999Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4437095Z Ran 1 test in 0.004s 2022-11-23T02:49:35.4437101Z 2022-11-23T02:49:35.4437197Z OK (skipped=1) 2022-11-23T02:49:35.4437202Z 2022-11-23T02:49:35.4437318Z Generating XML reports... 2022-11-23T02:49:35.4437758Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023044.xml 2022-11-23T02:49:35.4438070Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4438437Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4438659Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4439039Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4439216Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4439222Z 2022-11-23T02:49:35.4439319Z Running tests... 2022-11-23T02:49:35.4439580Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4439879Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69007 2022-11-23T02:49:35.4440086Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69008 2022-11-23T02:49:35.4440328Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4440742Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4440907Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4441288Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4441465Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4441686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4442053Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4442214Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4442589Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4442762Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4442989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4443385Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4443773Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4443984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4444197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4444424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp25ahuwcw 2022-11-23T02:49:35.4444669Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp25ahuwcw/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4444904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuofgwc0w 2022-11-23T02:49:35.4445153Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuofgwc0w/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4446083Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:49:35.4446988Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:49:35.4448227Z /opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1122.) 2022-11-23T02:49:35.4448506Z Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2022-11-23T02:49:35.4448720Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4449987Z /opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1122.) 2022-11-23T02:49:35.4450205Z Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2022-11-23T02:49:35.4450419Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4451325Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:49:35.4452240Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:49:35.4453148Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:49:35.4454044Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:49:35.4454952Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:49:35.4455853Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:49:35.4456765Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:49:35.4457712Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:49:35.4458608Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:49:35.4459551Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-11-23T02:49:35.4459648Z ok (4.969s) 2022-11-23T02:49:35.4459654Z 2022-11-23T02:49:35.4459922Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4460019Z Ran 1 test in 4.970s 2022-11-23T02:49:35.4460025Z 2022-11-23T02:49:35.4460107Z OK 2022-11-23T02:49:35.4460113Z 2022-11-23T02:49:35.4460224Z Generating XML reports... 2022-11-23T02:49:35.4460663Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023048.xml 2022-11-23T02:49:35.4460976Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4461342Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4461504Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4461888Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4462063Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4462069Z 2022-11-23T02:49:35.4462170Z Running tests... 2022-11-23T02:49:35.4462434Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4462841Z test_ddp_device (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.005s) 2022-11-23T02:49:35.4462848Z 2022-11-23T02:49:35.4463107Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4463206Z Ran 1 test in 0.005s 2022-11-23T02:49:35.4463214Z 2022-11-23T02:49:35.4463312Z OK (skipped=1) 2022-11-23T02:49:35.4463318Z 2022-11-23T02:49:35.4463431Z Generating XML reports... 2022-11-23T02:49:35.4463868Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023057.xml 2022-11-23T02:49:35.4464179Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4464547Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4464708Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4465088Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4465266Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4465272Z 2022-11-23T02:49:35.4465368Z Running tests... 2022-11-23T02:49:35.4465634Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4466128Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.003s) 2022-11-23T02:49:35.4466135Z 2022-11-23T02:49:35.4466395Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4466491Z Ran 1 test in 0.003s 2022-11-23T02:49:35.4466497Z 2022-11-23T02:49:35.4466589Z OK (skipped=1) 2022-11-23T02:49:35.4466595Z 2022-11-23T02:49:35.4466711Z Generating XML reports... 2022-11-23T02:49:35.4467137Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023101.xml 2022-11-23T02:49:35.4467446Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4467812Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4468042Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4468430Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4468606Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4468612Z 2022-11-23T02:49:35.4468708Z Running tests... 2022-11-23T02:49:35.4468970Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4469874Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78685 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.636s) 2022-11-23T02:49:35.4469881Z 2022-11-23T02:49:35.4470146Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4470250Z Ran 1 test in 0.636s 2022-11-23T02:49:35.4470256Z 2022-11-23T02:49:35.4470350Z OK (skipped=1) 2022-11-23T02:49:35.4470357Z 2022-11-23T02:49:35.4470467Z Generating XML reports... 2022-11-23T02:49:35.4470905Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023106.xml 2022-11-23T02:49:35.4471215Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4471584Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4471747Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4472127Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4472304Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4472311Z 2022-11-23T02:49:35.4472410Z Running tests... 2022-11-23T02:49:35.4472677Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4473585Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.633s) 2022-11-23T02:49:35.4473592Z 2022-11-23T02:49:35.4473855Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4473953Z Ran 1 test in 0.633s 2022-11-23T02:49:35.4473959Z 2022-11-23T02:49:35.4474052Z OK (skipped=1) 2022-11-23T02:49:35.4474057Z 2022-11-23T02:49:35.4474171Z Generating XML reports... 2022-11-23T02:49:35.4474608Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023110.xml 2022-11-23T02:49:35.4474979Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4475340Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4475502Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4475879Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4476055Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4476060Z 2022-11-23T02:49:35.4476157Z Running tests... 2022-11-23T02:49:35.4476420Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4476748Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69482 2022-11-23T02:49:35.4477003Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69483 2022-11-23T02:49:35.4477263Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4477635Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4477800Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4478181Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4478357Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4478579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4478977Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4479352Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4479513Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4479894Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4480069Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4480291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4480685Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4480896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4481112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4481326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:49:35.4481551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:49:35.4481943Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.4482332Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.4482567Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf651t9xy 2022-11-23T02:49:35.4482811Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf651t9xy/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4483041Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbbmi2wo2 2022-11-23T02:49:35.4483291Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbbmi2wo2/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4483565Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4483777Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4483994Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4484204Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4484297Z ok (7.983s) 2022-11-23T02:49:35.4484303Z 2022-11-23T02:49:35.4484572Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4484670Z Ran 1 test in 7.984s 2022-11-23T02:49:35.4484676Z 2022-11-23T02:49:35.4484759Z OK 2022-11-23T02:49:35.4484764Z 2022-11-23T02:49:35.4484878Z Generating XML reports... 2022-11-23T02:49:35.4485313Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023115.xml 2022-11-23T02:49:35.4485673Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4486048Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4486211Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4486590Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4486758Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4486773Z 2022-11-23T02:49:35.4486860Z Running tests... 2022-11-23T02:49:35.4487123Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4487438Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69705 2022-11-23T02:49:35.4487640Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69706 2022-11-23T02:49:35.4487946Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4488313Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4488478Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4488859Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4489035Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4489260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4489653Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4490023Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4490187Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4490564Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4490744Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4490967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4491360Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4491574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4491832Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-11-23T02:49:35.4492043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4492368Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-11-23T02:49:35.4492600Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe7emsqn5 2022-11-23T02:49:35.4492847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe7emsqn5/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4493069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg_7h51ut 2022-11-23T02:49:35.4493313Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg_7h51ut/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4493529Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4493740Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4493954Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4494214Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4494469Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-11-23T02:49:35.4494723Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-11-23T02:49:35.4494979Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-11-23T02:49:35.4495236Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-11-23T02:49:35.4495451Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4495662Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4495881Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4496093Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4496348Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-11-23T02:49:35.4496605Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-11-23T02:49:35.4496860Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations 2022-11-23T02:49:35.4497112Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations 2022-11-23T02:49:35.4497328Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4497540Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4497761Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4497974Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4498054Z ok (8.769s) 2022-11-23T02:49:35.4498071Z 2022-11-23T02:49:35.4498337Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4498433Z Ran 1 test in 8.770s 2022-11-23T02:49:35.4498439Z 2022-11-23T02:49:35.4498519Z OK 2022-11-23T02:49:35.4498525Z 2022-11-23T02:49:35.4498638Z Generating XML reports... 2022-11-23T02:49:35.4499079Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023127.xml 2022-11-23T02:49:35.4499389Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4499759Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4499927Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4500375Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4500551Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4500557Z 2022-11-23T02:49:35.4500652Z Running tests... 2022-11-23T02:49:35.4500915Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4501832Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77378 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.639s) 2022-11-23T02:49:35.4502503Z 2022-11-23T02:49:35.4502780Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4503114Z Ran 1 test in 0.639s 2022-11-23T02:49:35.4503316Z 2022-11-23T02:49:35.4503437Z OK (skipped=1) 2022-11-23T02:49:35.4503584Z 2022-11-23T02:49:35.4503696Z Generating XML reports... 2022-11-23T02:49:35.4504331Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023140.xml 2022-11-23T02:49:35.4504981Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4505599Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4506036Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4506618Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4507085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4507311Z 2022-11-23T02:49:35.4507408Z Running tests... 2022-11-23T02:49:35.4507814Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4508337Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69988 2022-11-23T02:49:35.4508874Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69989 2022-11-23T02:49:35.4509367Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4510033Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4510468Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4511049Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4511511Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4511936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4512592Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4513256Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4513688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4514275Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4514725Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4515155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4515809Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4516369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4517132Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:49:35.4517872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4518639Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:49:35.4519499Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnj4epsjt 2022-11-23T02:49:35.4520044Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnj4epsjt/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4520559Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1rrgfeg8 2022-11-23T02:49:35.4521063Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1rrgfeg8/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4521549Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4522006Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4522498Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations. 2022-11-23T02:49:35.4523041Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations. 2022-11-23T02:49:35.4523611Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created. 2022-11-23T02:49:35.4524256Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0 2022-11-23T02:49:35.4524869Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created. 2022-11-23T02:49:35.4525483Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively. 2022-11-23T02:49:35.4526141Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0 2022-11-23T02:49:35.4526778Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively. 2022-11-23T02:49:35.4527346Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4527936Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4528264Z ok (7.580s) 2022-11-23T02:49:35.4528398Z 2022-11-23T02:49:35.4528684Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4528984Z Ran 1 test in 7.581s 2022-11-23T02:49:35.4529133Z 2022-11-23T02:49:35.4529216Z OK 2022-11-23T02:49:35.4529337Z 2022-11-23T02:49:35.4529449Z Generating XML reports... 2022-11-23T02:49:35.4530040Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023145.xml 2022-11-23T02:49:35.4530679Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4531294Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4531807Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4532382Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4532834Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4533047Z 2022-11-23T02:49:35.4533143Z Running tests... 2022-11-23T02:49:35.4533544Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4534103Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70205 2022-11-23T02:49:35.4534676Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70206 2022-11-23T02:49:35.4535160Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4535862Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4536297Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4536878Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4537328Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4537755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4538395Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4539046Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4539476Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4540046Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4540500Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4540930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4541563Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4542057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4542509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4542998Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4543503Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp74rz16_j 2022-11-23T02:49:35.4544037Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp74rz16_j/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4544560Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4545058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeu_a477q 2022-11-23T02:49:35.4545556Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeu_a477q/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4546034Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4546487Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4546944Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4547443Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4547815Z ok (7.913s) 2022-11-23T02:49:35.4548018Z 2022-11-23T02:49:35.4548307Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4548619Z Ran 1 test in 7.914s 2022-11-23T02:49:35.4548766Z 2022-11-23T02:49:35.4548847Z OK 2022-11-23T02:49:35.4548969Z 2022-11-23T02:49:35.4549084Z Generating XML reports... 2022-11-23T02:49:35.4549665Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023157.xml 2022-11-23T02:49:35.4550304Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4550975Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4551431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4552026Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4552541Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4552763Z 2022-11-23T02:49:35.4552874Z Running tests... 2022-11-23T02:49:35.4553299Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4553855Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70484 2022-11-23T02:49:35.4554423Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70485 2022-11-23T02:49:35.4554899Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4555543Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4555977Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4556556Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4557005Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4557433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4558078Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4558715Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4559145Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4559716Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4560165Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4560600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4561249Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4561737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4562188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4562671Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4563176Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0vessx5a 2022-11-23T02:49:35.4563684Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0vessx5a/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4564207Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4564714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8t9di34o 2022-11-23T02:49:35.4565300Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8t9di34o/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4565786Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4566248Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4566695Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4567152Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4567499Z ok (8.095s) 2022-11-23T02:49:35.4567634Z 2022-11-23T02:49:35.4567958Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4568270Z Ran 1 test in 8.096s 2022-11-23T02:49:35.4568417Z 2022-11-23T02:49:35.4568499Z OK 2022-11-23T02:49:35.4568620Z 2022-11-23T02:49:35.4568728Z Generating XML reports... 2022-11-23T02:49:35.4569379Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023209.xml 2022-11-23T02:49:35.4570026Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4570777Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4571220Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4571827Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4572294Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4572507Z 2022-11-23T02:49:35.4572603Z Running tests... 2022-11-23T02:49:35.4573004Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4573628Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70763 2022-11-23T02:49:35.4574257Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70764 2022-11-23T02:49:35.4574733Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4575370Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4575799Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4576372Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4576821Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4577248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4577893Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4578542Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4578965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4579537Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4579988Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4580414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4581052Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4581550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4582073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4582558Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4583063Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk0nhddtd 2022-11-23T02:49:35.4583571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk0nhddtd/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4584101Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4584608Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp096rag_n 2022-11-23T02:49:35.4585125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp096rag_n/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4585665Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4586127Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4586571Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4587030Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4587429Z ok (7.870s) 2022-11-23T02:49:35.4587574Z 2022-11-23T02:49:35.4587880Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4588212Z Ran 1 test in 7.871s 2022-11-23T02:49:35.4588359Z 2022-11-23T02:49:35.4588439Z OK 2022-11-23T02:49:35.4588561Z 2022-11-23T02:49:35.4588674Z Generating XML reports... 2022-11-23T02:49:35.4589308Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023221.xml 2022-11-23T02:49:35.4589945Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4590558Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4590991Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4591560Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4592008Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4592224Z 2022-11-23T02:49:35.4592347Z Running tests... 2022-11-23T02:49:35.4592800Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4593423Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71042 2022-11-23T02:49:35.4594059Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71043 2022-11-23T02:49:35.4594553Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4595184Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4595643Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4596251Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4596699Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4597143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4597797Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4598326Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4598908Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4599361Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4599825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4600481Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4601156Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4601666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4602118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4602661Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4603182Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4603673Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx5ykwt17 2022-11-23T02:49:35.4604180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx5ykwt17/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4604681Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt06p70fl 2022-11-23T02:49:35.4605186Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt06p70fl/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4605672Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4606126Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4606588Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4607037Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4607371Z ok (7.877s) 2022-11-23T02:49:35.4607507Z 2022-11-23T02:49:35.4607827Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4608139Z Ran 1 test in 7.877s 2022-11-23T02:49:35.4608289Z 2022-11-23T02:49:35.4608371Z OK 2022-11-23T02:49:35.4608493Z 2022-11-23T02:49:35.4608605Z Generating XML reports... 2022-11-23T02:49:35.4609196Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023233.xml 2022-11-23T02:49:35.4609827Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4610440Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4610887Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4611471Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4611920Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4612134Z 2022-11-23T02:49:35.4612230Z Running tests... 2022-11-23T02:49:35.4612634Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4613238Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71321 2022-11-23T02:49:35.4613860Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71322 2022-11-23T02:49:35.4614342Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4615063Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4615493Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4616066Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4616518Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4616948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4617583Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4618236Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4618669Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4619300Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4619756Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4620190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4620831Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4621327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4621770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4622257Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4622769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8yvb85zk 2022-11-23T02:49:35.4623279Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8yvb85zk/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4623801Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4624309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcsmz7pz8 2022-11-23T02:49:35.4624812Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcsmz7pz8/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4625297Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4625748Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4626205Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4626657Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4626991Z ok (7.973s) 2022-11-23T02:49:35.4627129Z 2022-11-23T02:49:35.4627400Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4627711Z Ran 1 test in 7.973s 2022-11-23T02:49:35.4627861Z 2022-11-23T02:49:35.4627933Z OK 2022-11-23T02:49:35.4628053Z 2022-11-23T02:49:35.4628166Z Generating XML reports... 2022-11-23T02:49:35.4628761Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023245.xml 2022-11-23T02:49:35.4629402Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4630018Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4630450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4631028Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4631541Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4631744Z 2022-11-23T02:49:35.4631840Z Running tests... 2022-11-23T02:49:35.4632243Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4632858Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71600 2022-11-23T02:49:35.4633476Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71601 2022-11-23T02:49:35.4633958Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4634602Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4635036Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4635665Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4636112Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4636547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4637168Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4637595Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4638164Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4638611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4639040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4639679Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4640348Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4640839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4641294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4641784Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4642294Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnriyb_ff 2022-11-23T02:49:35.4642799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnriyb_ff/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4643320Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4643820Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5wf99fzl 2022-11-23T02:49:35.4644321Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5wf99fzl/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4644801Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4645257Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4645713Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4646165Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4646498Z ok (8.182s) 2022-11-23T02:49:35.4646634Z 2022-11-23T02:49:35.4646894Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4647207Z Ran 1 test in 8.182s 2022-11-23T02:49:35.4647356Z 2022-11-23T02:49:35.4647499Z OK 2022-11-23T02:49:35.4647624Z 2022-11-23T02:49:35.4647855Z Generating XML reports... 2022-11-23T02:49:35.4648459Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023257.xml 2022-11-23T02:49:35.4649095Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4649705Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4650122Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4650692Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4651141Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4651356Z 2022-11-23T02:49:35.4651451Z Running tests... 2022-11-23T02:49:35.4651851Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4652535Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71879 2022-11-23T02:49:35.4653161Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71880 2022-11-23T02:49:35.4653645Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4654289Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4654717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4655287Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4655736Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4656171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4656812Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4657463Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4657880Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4658454Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4658902Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4659328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4659971Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4660463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4660914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4661408Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4661905Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy4h80vly 2022-11-23T02:49:35.4662414Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy4h80vly/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4662932Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4663439Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphpuivyqs 2022-11-23T02:49:35.4663950Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphpuivyqs/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4664504Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4664959Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4665406Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4665863Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4666189Z ok (8.473s) 2022-11-23T02:49:35.4666323Z 2022-11-23T02:49:35.4666595Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4666907Z Ran 1 test in 8.473s 2022-11-23T02:49:35.4667060Z 2022-11-23T02:49:35.4667141Z OK 2022-11-23T02:49:35.4667262Z 2022-11-23T02:49:35.4667376Z Generating XML reports... 2022-11-23T02:49:35.4668011Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023309.xml 2022-11-23T02:49:35.4668658Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4669272Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4669703Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4670278Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4670726Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4670940Z 2022-11-23T02:49:35.4671036Z Running tests... 2022-11-23T02:49:35.4671432Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4672047Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72158 2022-11-23T02:49:35.4672673Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72159 2022-11-23T02:49:35.4673157Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4673805Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4674236Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4674808Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4675257Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4675675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4676295Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4676730Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4677302Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4677749Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4678176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4678817Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4679483Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4679965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4680425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4680979Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4681486Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp86jxmpc5 2022-11-23T02:49:35.4681996Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp86jxmpc5/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4682516Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4683020Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg3tlju97 2022-11-23T02:49:35.4683525Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg3tlju97/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4684002Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4684522Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4684989Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4685444Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4685777Z ok (8.065s) 2022-11-23T02:49:35.4685912Z 2022-11-23T02:49:35.4686188Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4686499Z Ran 1 test in 8.066s 2022-11-23T02:49:35.4686638Z 2022-11-23T02:49:35.4686720Z OK 2022-11-23T02:49:35.4686842Z 2022-11-23T02:49:35.4686955Z Generating XML reports... 2022-11-23T02:49:35.4687550Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023322.xml 2022-11-23T02:49:35.4688237Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4688861Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4689297Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4689867Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4690313Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4690527Z 2022-11-23T02:49:35.4690624Z Running tests... 2022-11-23T02:49:35.4691026Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4691636Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72437 2022-11-23T02:49:35.4692250Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72438 2022-11-23T02:49:35.4692739Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4693386Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4693816Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4694379Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4694830Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4695257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4695896Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4696551Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4696983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4697634Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4698075Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4698498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4699139Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4699631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4700083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4700575Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4701134Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3fqwl5zf 2022-11-23T02:49:35.4701647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3fqwl5zf/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4702156Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4702661Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf6bucyxe 2022-11-23T02:49:35.4703167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf6bucyxe/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4703654Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4704109Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4704566Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4705026Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4705353Z ok (7.868s) 2022-11-23T02:49:35.4705487Z 2022-11-23T02:49:35.4705761Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4706075Z Ran 1 test in 7.868s 2022-11-23T02:49:35.4706224Z 2022-11-23T02:49:35.4706308Z OK 2022-11-23T02:49:35.4706429Z 2022-11-23T02:49:35.4706542Z Generating XML reports... 2022-11-23T02:49:35.4707133Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023334.xml 2022-11-23T02:49:35.4707770Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4708377Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4708812Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4709386Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4709843Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4710057Z 2022-11-23T02:49:35.4710155Z Running tests... 2022-11-23T02:49:35.4710560Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4711173Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72716 2022-11-23T02:49:35.4711790Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72717 2022-11-23T02:49:35.4712269Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4712913Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4713411Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4713985Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4714437Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4714864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4715504Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4716147Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4716577Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4717152Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4717656Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4718085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4718730Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4719228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4719679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4720159Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4720662Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptmmlfh22 2022-11-23T02:49:35.4721169Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptmmlfh22/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4721700Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4722206Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ypzsh4k 2022-11-23T02:49:35.4722714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ypzsh4k/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4723200Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4723657Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4724105Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4724557Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4724891Z ok (8.057s) 2022-11-23T02:49:35.4725030Z 2022-11-23T02:49:35.4725298Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4725621Z Ran 1 test in 8.058s 2022-11-23T02:49:35.4725770Z 2022-11-23T02:49:35.4725850Z OK 2022-11-23T02:49:35.4725971Z 2022-11-23T02:49:35.4726075Z Generating XML reports... 2022-11-23T02:49:35.4726669Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023346.xml 2022-11-23T02:49:35.4727306Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4727967Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4728399Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4728974Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4729422Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4729636Z 2022-11-23T02:49:35.4729734Z Running tests... 2022-11-23T02:49:35.4730207Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4730760Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72995 2022-11-23T02:49:35.4731329Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72996 2022-11-23T02:49:35.4731808Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4732455Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4732886Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4733457Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4733950Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4734389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4735012Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4735443Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4736013Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4736464Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4736889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4737535Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4738203Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4738699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4739156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4739645Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4740150Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppvckd85j 2022-11-23T02:49:35.4740659Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppvckd85j/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4741183Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4741685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3sz6fxmv 2022-11-23T02:49:35.4742184Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3sz6fxmv/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4742673Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4743132Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4743591Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4744045Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4744377Z ok (7.885s) 2022-11-23T02:49:35.4744512Z 2022-11-23T02:49:35.4744785Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4745086Z Ran 1 test in 7.886s 2022-11-23T02:49:35.4745235Z 2022-11-23T02:49:35.4745319Z OK 2022-11-23T02:49:35.4745440Z 2022-11-23T02:49:35.4745552Z Generating XML reports... 2022-11-23T02:49:35.4746149Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023359.xml 2022-11-23T02:49:35.4746855Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4747470Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4747905Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4748467Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4748919Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4749136Z 2022-11-23T02:49:35.4749236Z Running tests... 2022-11-23T02:49:35.4749640Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4750323Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73274 2022-11-23T02:49:35.4750540Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73275 2022-11-23T02:49:35.4750791Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4751166Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4751329Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4751708Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4751886Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4752109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4752498Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4752869Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4753036Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4753416Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4753596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4753816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4754209Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4754422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4754640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4754903Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4755137Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmaqeyyvh 2022-11-23T02:49:35.4755384Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmaqeyyvh/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4755634Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4755870Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkvklznch 2022-11-23T02:49:35.4756117Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkvklznch/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4756333Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4756554Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4756825Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4757034Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4757127Z ok (7.891s) 2022-11-23T02:49:35.4757132Z 2022-11-23T02:49:35.4757401Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4757501Z Ran 1 test in 7.891s 2022-11-23T02:49:35.4757507Z 2022-11-23T02:49:35.4757579Z OK 2022-11-23T02:49:35.4757593Z 2022-11-23T02:49:35.4757697Z Generating XML reports... 2022-11-23T02:49:35.4758134Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023411.xml 2022-11-23T02:49:35.4758449Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4758866Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4759033Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4759416Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4759594Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4759601Z 2022-11-23T02:49:35.4759700Z Running tests... 2022-11-23T02:49:35.4759964Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4760393Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'ucc', 'gloo'} (0.002s) 2022-11-23T02:49:35.4760400Z 2022-11-23T02:49:35.4760660Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4760757Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4760763Z 2022-11-23T02:49:35.4760858Z OK (skipped=1) 2022-11-23T02:49:35.4760864Z 2022-11-23T02:49:35.4760984Z Generating XML reports... 2022-11-23T02:49:35.4761425Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023423.xml 2022-11-23T02:49:35.4761735Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4762104Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4762268Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4762648Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4762824Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4762830Z 2022-11-23T02:49:35.4762928Z Running tests... 2022-11-23T02:49:35.4763194Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4763482Z test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73619 2022-11-23T02:49:35.4763692Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73620 2022-11-23T02:49:35.4763941Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4764311Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4764474Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4764854Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4765032Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4765256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4765721Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4766088Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4766249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4766627Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4766803Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4767027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4767419Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4767628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4768019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4768260Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc7ml3d5p 2022-11-23T02:49:35.4768506Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc7ml3d5p/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4768741Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjtoobj5r 2022-11-23T02:49:35.4768989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjtoobj5r/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4769081Z ok (7.320s) 2022-11-23T02:49:35.4769087Z 2022-11-23T02:49:35.4769363Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4769452Z Ran 1 test in 7.320s 2022-11-23T02:49:35.4769457Z 2022-11-23T02:49:35.4769542Z OK 2022-11-23T02:49:35.4769547Z 2022-11-23T02:49:35.4769660Z Generating XML reports... 2022-11-23T02:49:35.4770104Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023427.xml 2022-11-23T02:49:35.4770414Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4770784Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4770944Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4771324Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4771498Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4771504Z 2022-11-23T02:49:35.4771600Z Running tests... 2022-11-23T02:49:35.4771867Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4772185Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73829 2022-11-23T02:49:35.4772394Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73830 2022-11-23T02:49:35.4772646Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4773013Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4773176Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4773556Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4773732Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4773956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4774324Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4774551Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4774933Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4775100Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4775321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4775713Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4776101Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4776311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4776580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4776821Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc3yf9rvi 2022-11-23T02:49:35.4777069Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc3yf9rvi/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4777303Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3aqgqusv 2022-11-23T02:49:35.4777554Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3aqgqusv/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4777772Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4777985Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4778383Z /opt/conda/lib/python3.8/tempfile.py:818: ResourceWarning: Implicitly cleaning up 2022-11-23T02:49:35.4778541Z _warnings.warn(warn_message, ResourceWarning) 2022-11-23T02:49:35.4778941Z /opt/conda/lib/python3.8/tempfile.py:818: ResourceWarning: Implicitly cleaning up 2022-11-23T02:49:35.4779092Z _warnings.warn(warn_message, ResourceWarning) 2022-11-23T02:49:35.4779182Z ok (7.417s) 2022-11-23T02:49:35.4779188Z 2022-11-23T02:49:35.4779452Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4779550Z Ran 1 test in 7.417s 2022-11-23T02:49:35.4779556Z 2022-11-23T02:49:35.4779638Z OK 2022-11-23T02:49:35.4779643Z 2022-11-23T02:49:35.4779755Z Generating XML reports... 2022-11-23T02:49:35.4780183Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023438.xml 2022-11-23T02:49:35.4780496Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4780865Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4781035Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4781415Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4781591Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4781596Z 2022-11-23T02:49:35.4781694Z Running tests... 2022-11-23T02:49:35.4781957Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4782260Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74046 2022-11-23T02:49:35.4782470Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74047 2022-11-23T02:49:35.4782719Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4783091Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4783331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4783719Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4783895Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4784119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4784510Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4784876Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4785038Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4785467Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4785651Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4785878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4786273Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4786478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4786711Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr20vkw79 2022-11-23T02:49:35.4786956Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr20vkw79/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4787170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4787406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyvd9cldj 2022-11-23T02:49:35.4787656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyvd9cldj/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4787874Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4788090Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4788180Z ok (5.228s) 2022-11-23T02:49:35.4788186Z 2022-11-23T02:49:35.4788455Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4788555Z Ran 1 test in 5.229s 2022-11-23T02:49:35.4788561Z 2022-11-23T02:49:35.4788643Z OK 2022-11-23T02:49:35.4788649Z 2022-11-23T02:49:35.4788760Z Generating XML reports... 2022-11-23T02:49:35.4789202Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023450.xml 2022-11-23T02:49:35.4789515Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4789895Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4790056Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4790439Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4790617Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4790623Z 2022-11-23T02:49:35.4790725Z Running tests... 2022-11-23T02:49:35.4790992Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4791295Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74319 2022-11-23T02:49:35.4791489Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74320 2022-11-23T02:49:35.4791745Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4792176Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4792339Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4792718Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4792894Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4793120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4793514Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4793887Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4794106Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4794494Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4794668Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4794890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4795283Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4795495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4795705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4795934Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6o0wd0vx 2022-11-23T02:49:35.4796187Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6o0wd0vx/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4796424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpznq6icig 2022-11-23T02:49:35.4796669Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpznq6icig/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4796888Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4797103Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4797184Z ok (7.637s) 2022-11-23T02:49:35.4797205Z 2022-11-23T02:49:35.4797462Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4797581Z Ran 1 test in 7.637s 2022-11-23T02:49:35.4797587Z 2022-11-23T02:49:35.4797721Z OK 2022-11-23T02:49:35.4797727Z 2022-11-23T02:49:35.4797853Z Generating XML reports... 2022-11-23T02:49:35.4798286Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023459.xml 2022-11-23T02:49:35.4798601Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4798969Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4799132Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4799513Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4799691Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4799697Z 2022-11-23T02:49:35.4799799Z Running tests... 2022-11-23T02:49:35.4800074Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4800543Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.002s) 2022-11-23T02:49:35.4800609Z 2022-11-23T02:49:35.4800877Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4800978Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4800984Z 2022-11-23T02:49:35.4801082Z OK (skipped=1) 2022-11-23T02:49:35.4801087Z 2022-11-23T02:49:35.4801206Z Generating XML reports... 2022-11-23T02:49:35.4801647Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023511.xml 2022-11-23T02:49:35.4801964Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4802336Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4802504Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4802885Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4803110Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4803117Z 2022-11-23T02:49:35.4803218Z Running tests... 2022-11-23T02:49:35.4803484Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4803939Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'ucc', 'gloo'} (0.002s) 2022-11-23T02:49:35.4803946Z 2022-11-23T02:49:35.4804213Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4804317Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4804323Z 2022-11-23T02:49:35.4804408Z OK (skipped=1) 2022-11-23T02:49:35.4804427Z 2022-11-23T02:49:35.4804530Z Generating XML reports... 2022-11-23T02:49:35.4804968Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023515.xml 2022-11-23T02:49:35.4805288Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4805664Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4805828Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4806205Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4806384Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4806390Z 2022-11-23T02:49:35.4806488Z Running tests... 2022-11-23T02:49:35.4806756Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4807255Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.002s) 2022-11-23T02:49:35.4807262Z 2022-11-23T02:49:35.4807524Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4807629Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4807635Z 2022-11-23T02:49:35.4807778Z OK (skipped=1) 2022-11-23T02:49:35.4807783Z 2022-11-23T02:49:35.4807900Z Generating XML reports... 2022-11-23T02:49:35.4808337Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023519.xml 2022-11-23T02:49:35.4808649Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4809021Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4809184Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4809572Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4809752Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4809818Z 2022-11-23T02:49:35.4809933Z Running tests... 2022-11-23T02:49:35.4810206Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4810665Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.002s) 2022-11-23T02:49:35.4810685Z 2022-11-23T02:49:35.4810937Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4811040Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4811045Z 2022-11-23T02:49:35.4811140Z OK (skipped=1) 2022-11-23T02:49:35.4811146Z 2022-11-23T02:49:35.4811259Z Generating XML reports... 2022-11-23T02:49:35.4811700Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023523.xml 2022-11-23T02:49:35.4812014Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4812438Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4812609Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4812990Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4813170Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4813176Z 2022-11-23T02:49:35.4813276Z Running tests... 2022-11-23T02:49:35.4813538Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4813955Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.003s) 2022-11-23T02:49:35.4813962Z 2022-11-23T02:49:35.4814230Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4814332Z Ran 1 test in 0.003s 2022-11-23T02:49:35.4814338Z 2022-11-23T02:49:35.4814433Z OK (skipped=1) 2022-11-23T02:49:35.4814443Z 2022-11-23T02:49:35.4814561Z Generating XML reports... 2022-11-23T02:49:35.4815005Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023527.xml 2022-11-23T02:49:35.4815317Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4815696Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4815865Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4816249Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4816413Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4816434Z 2022-11-23T02:49:35.4816522Z Running tests... 2022-11-23T02:49:35.4816792Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4817106Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74866 2022-11-23T02:49:35.4817319Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74867 2022-11-23T02:49:35.4817572Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4817946Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4818109Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4818497Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4818678Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4818907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4819332Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4819496Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4819880Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4820060Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4820292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4820691Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4821088Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4821350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4821575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4821810Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprmvays7s 2022-11-23T02:49:35.4822057Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprmvays7s/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4822290Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi8itej6j 2022-11-23T02:49:35.4822526Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi8itej6j/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4823299Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:49:35.4824053Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:49:35.4824278Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4824495Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4824593Z ok (7.429s) 2022-11-23T02:49:35.4824603Z 2022-11-23T02:49:35.4824876Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4824978Z Ran 1 test in 7.430s 2022-11-23T02:49:35.4824987Z 2022-11-23T02:49:35.4825076Z OK 2022-11-23T02:49:35.4825082Z 2022-11-23T02:49:35.4825185Z Generating XML reports... 2022-11-23T02:49:35.4825624Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023532.xml 2022-11-23T02:49:35.4825944Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4826320Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4826485Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4826872Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4827109Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4827116Z 2022-11-23T02:49:35.4827214Z Running tests... 2022-11-23T02:49:35.4827485Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4828416Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78338 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.585s) 2022-11-23T02:49:35.4828423Z 2022-11-23T02:49:35.4828685Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4828788Z Ran 1 test in 0.585s 2022-11-23T02:49:35.4828793Z 2022-11-23T02:49:35.4828892Z OK (skipped=1) 2022-11-23T02:49:35.4828897Z 2022-11-23T02:49:35.4829019Z Generating XML reports... 2022-11-23T02:49:35.4829507Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023543.xml 2022-11-23T02:49:35.4829825Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4830201Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4830368Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4830754Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4830931Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4830938Z 2022-11-23T02:49:35.4831041Z Running tests... 2022-11-23T02:49:35.4831311Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4831771Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.002s) 2022-11-23T02:49:35.4831780Z 2022-11-23T02:49:35.4832043Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4832143Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4832148Z 2022-11-23T02:49:35.4832246Z OK (skipped=1) 2022-11-23T02:49:35.4832251Z 2022-11-23T02:49:35.4832354Z Generating XML reports... 2022-11-23T02:49:35.4832794Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023548.xml 2022-11-23T02:49:35.4833111Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4833486Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4833651Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4834035Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4834218Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4834224Z 2022-11-23T02:49:35.4834324Z Running tests... 2022-11-23T02:49:35.4834594Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4835039Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.002s) 2022-11-23T02:49:35.4835047Z 2022-11-23T02:49:35.4835313Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4835418Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4835424Z 2022-11-23T02:49:35.4835520Z OK (skipped=1) 2022-11-23T02:49:35.4835526Z 2022-11-23T02:49:35.4835645Z Generating XML reports... 2022-11-23T02:49:35.4836083Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023552.xml 2022-11-23T02:49:35.4836456Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4836832Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4836999Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4837380Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4837565Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4837572Z 2022-11-23T02:49:35.4837673Z Running tests... 2022-11-23T02:49:35.4837942Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4838255Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75281 2022-11-23T02:49:35.4838494Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75282 2022-11-23T02:49:35.4838754Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4839127Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4839296Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4839677Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4839859Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4840086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4840458Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4840627Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4841007Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4841186Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4841411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4841807Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4842201Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4842419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4842636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4842877Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn5fivo4u 2022-11-23T02:49:35.4843138Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn5fivo4u/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4843373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf1_dlodd 2022-11-23T02:49:35.4843622Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf1_dlodd/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4843713Z ok (5.274s) 2022-11-23T02:49:35.4843719Z 2022-11-23T02:49:35.4843986Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4844088Z Ran 1 test in 5.275s 2022-11-23T02:49:35.4844094Z 2022-11-23T02:49:35.4844165Z OK 2022-11-23T02:49:35.4844187Z 2022-11-23T02:49:35.4844289Z Generating XML reports... 2022-11-23T02:49:35.4844729Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023556.xml 2022-11-23T02:49:35.4845046Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4845474Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4845637Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4846019Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4846202Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4846208Z 2022-11-23T02:49:35.4846308Z Running tests... 2022-11-23T02:49:35.4846575Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4847552Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78595 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.588s) 2022-11-23T02:49:35.4847565Z 2022-11-23T02:49:35.4847883Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4847986Z Ran 1 test in 0.588s 2022-11-23T02:49:35.4847992Z 2022-11-23T02:49:35.4848090Z OK (skipped=1) 2022-11-23T02:49:35.4848095Z 2022-11-23T02:49:35.4848208Z Generating XML reports... 2022-11-23T02:49:35.4848654Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023606.xml 2022-11-23T02:49:35.4848972Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4849348Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4849513Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4849903Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4850089Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4850095Z 2022-11-23T02:49:35.4850199Z Running tests... 2022-11-23T02:49:35.4850470Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4850932Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'ucc', 'gloo'} (0.003s) 2022-11-23T02:49:35.4850938Z 2022-11-23T02:49:35.4851203Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4851304Z Ran 1 test in 0.003s 2022-11-23T02:49:35.4851310Z 2022-11-23T02:49:35.4851394Z OK (skipped=1) 2022-11-23T02:49:35.4851416Z 2022-11-23T02:49:35.4851518Z Generating XML reports... 2022-11-23T02:49:35.4851957Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023610.xml 2022-11-23T02:49:35.4852275Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4852647Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4852811Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4853190Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4853376Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4853382Z 2022-11-23T02:49:35.4853487Z Running tests... 2022-11-23T02:49:35.4853757Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4854681Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.587s) 2022-11-23T02:49:35.4854751Z 2022-11-23T02:49:35.4855023Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4855129Z Ran 1 test in 0.587s 2022-11-23T02:49:35.4855135Z 2022-11-23T02:49:35.4855230Z OK (skipped=1) 2022-11-23T02:49:35.4855236Z 2022-11-23T02:49:35.4855354Z Generating XML reports... 2022-11-23T02:49:35.4855794Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023615.xml 2022-11-23T02:49:35.4856111Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4856482Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4856649Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4857087Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4857267Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4857273Z 2022-11-23T02:49:35.4857374Z Running tests... 2022-11-23T02:49:35.4857649Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4857962Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75686 2022-11-23T02:49:35.4858169Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75687 2022-11-23T02:49:35.4858410Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4858786Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4858962Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4859353Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4859536Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4859761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4860132Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4860297Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4860682Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4860863Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4861094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4861493Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4861891Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4862107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4862321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4862557Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwa0d0cxr 2022-11-23T02:49:35.4862807Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwa0d0cxr/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4863042Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdr_kr6if 2022-11-23T02:49:35.4863354Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdr_kr6if/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4863699Z STAGE:2022-11-23 02:36:22 75687:75687 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4864029Z STAGE:2022-11-23 02:36:22 75686:75686 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4864255Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4864474Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:49:35.4864803Z STAGE:2022-11-23 02:36:23 75687:75687 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4865140Z STAGE:2022-11-23 02:36:23 75686:75686 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4865497Z STAGE:2022-11-23 02:36:23 75687:75687 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4865782Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 5 (function operator()) 2022-11-23T02:49:35.4866141Z STAGE:2022-11-23 02:36:23 75686:75686 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4866370Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 5 (function operator()) 2022-11-23T02:49:35.4866711Z STAGE:2022-11-23 02:36:23 75686:75686 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4867051Z STAGE:2022-11-23 02:36:23 75686:75686 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4867405Z STAGE:2022-11-23 02:36:23 75686:75686 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4867499Z ok (5.818s) 2022-11-23T02:49:35.4867506Z 2022-11-23T02:49:35.4867771Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4867873Z Ran 1 test in 5.819s 2022-11-23T02:49:35.4867879Z 2022-11-23T02:49:35.4867970Z OK 2022-11-23T02:49:35.4867980Z 2022-11-23T02:49:35.4868100Z Generating XML reports... 2022-11-23T02:49:35.4868545Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023619.xml 2022-11-23T02:49:35.4868854Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4869232Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4869397Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4869785Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4869963Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4869969Z 2022-11-23T02:49:35.4870073Z Running tests... 2022-11-23T02:49:35.4870343Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4870644Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75907 2022-11-23T02:49:35.4870852Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75908 2022-11-23T02:49:35.4871107Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4871480Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4871646Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4872031Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4872212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4872439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4872902Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4873273Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4873437Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4873820Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4874000Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4874226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4874626Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4874895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4875119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4875355Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5uztidz5 2022-11-23T02:49:35.4875606Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5uztidz5/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4875843Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppywoakij 2022-11-23T02:49:35.4876101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppywoakij/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4876201Z ok (5.529s) 2022-11-23T02:49:35.4876208Z 2022-11-23T02:49:35.4876481Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4876570Z Ran 1 test in 5.530s 2022-11-23T02:49:35.4876592Z 2022-11-23T02:49:35.4876665Z OK 2022-11-23T02:49:35.4876670Z 2022-11-23T02:49:35.4876796Z Generating XML reports... 2022-11-23T02:49:35.4877240Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023629.xml 2022-11-23T02:49:35.4877555Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4877930Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4878099Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4878482Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4878664Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4878670Z 2022-11-23T02:49:35.4878775Z Running tests... 2022-11-23T02:49:35.4879039Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4879360Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76118 2022-11-23T02:49:35.4879570Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76119 2022-11-23T02:49:35.4879828Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4880204Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4880369Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4880749Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4880930Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4881162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4881622Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4881993Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4882158Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4882541Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4882706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4882930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4883332Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4883549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4883819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4884061Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1qdp9a0g 2022-11-23T02:49:35.4884313Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1qdp9a0g/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4884549Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6dc86dew 2022-11-23T02:49:35.4884798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6dc86dew/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4884894Z ok (5.341s) 2022-11-23T02:49:35.4884901Z 2022-11-23T02:49:35.4885171Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4885273Z Ran 1 test in 5.341s 2022-11-23T02:49:35.4885279Z 2022-11-23T02:49:35.4885368Z OK 2022-11-23T02:49:35.4885374Z 2022-11-23T02:49:35.4885492Z Generating XML reports... 2022-11-23T02:49:35.4885941Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023639.xml 2022-11-23T02:49:35.4886257Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4886629Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4886800Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4887187Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4887369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4887375Z 2022-11-23T02:49:35.4887478Z Running tests... 2022-11-23T02:49:35.4887849Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4888786Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78684 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.588s) 2022-11-23T02:49:35.4888797Z 2022-11-23T02:49:35.4889064Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4889152Z Ran 1 test in 0.588s 2022-11-23T02:49:35.4889172Z 2022-11-23T02:49:35.4889257Z OK (skipped=1) 2022-11-23T02:49:35.4889262Z 2022-11-23T02:49:35.4889380Z Generating XML reports... 2022-11-23T02:49:35.4889818Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023649.xml 2022-11-23T02:49:35.4890132Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4890508Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4890744Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4891137Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4891322Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4891328Z 2022-11-23T02:49:35.4891433Z Running tests... 2022-11-23T02:49:35.4891704Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4892590Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.645s) 2022-11-23T02:49:35.4892598Z 2022-11-23T02:49:35.4892919Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4893032Z Ran 1 test in 0.645s 2022-11-23T02:49:35.4893038Z 2022-11-23T02:49:35.4893134Z OK (skipped=1) 2022-11-23T02:49:35.4893139Z 2022-11-23T02:49:35.4893256Z Generating XML reports... 2022-11-23T02:49:35.4893709Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023654.xml 2022-11-23T02:49:35.4894025Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4894399Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4894567Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4894953Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4895133Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4895145Z 2022-11-23T02:49:35.4895251Z Running tests... 2022-11-23T02:49:35.4895521Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4896462Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.658s) 2022-11-23T02:49:35.4896470Z 2022-11-23T02:49:35.4896737Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4896836Z Ran 1 test in 0.658s 2022-11-23T02:49:35.4896842Z 2022-11-23T02:49:35.4896942Z OK (skipped=1) 2022-11-23T02:49:35.4896948Z 2022-11-23T02:49:35.4897066Z Generating XML reports... 2022-11-23T02:49:35.4897497Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023659.xml 2022-11-23T02:49:35.4897815Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4898186Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4898353Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4898740Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4898918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4898924Z 2022-11-23T02:49:35.4899027Z Running tests... 2022-11-23T02:49:35.4899297Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4899782Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.003s) 2022-11-23T02:49:35.4899841Z 2022-11-23T02:49:35.4900109Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4900212Z Ran 1 test in 0.003s 2022-11-23T02:49:35.4900218Z 2022-11-23T02:49:35.4900318Z OK (skipped=1) 2022-11-23T02:49:35.4900324Z 2022-11-23T02:49:35.4900437Z Generating XML reports... 2022-11-23T02:49:35.4900884Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023703.xml 2022-11-23T02:49:35.4901199Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4901573Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4901739Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4902121Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4902348Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4902355Z 2022-11-23T02:49:35.4902463Z Running tests... 2022-11-23T02:49:35.4902735Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4903047Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76589 2022-11-23T02:49:35.4903256Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76590 2022-11-23T02:49:35.4903499Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4903869Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4904036Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4904423Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4904609Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4904837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4905239Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4905615Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4905780Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4906165Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4906346Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4906577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4906978Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4907195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4907406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4907788Z /opt/conda/lib/python3.8/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op 2022-11-23T02:49:35.4908045Z warnings.warn("Initializing zero-element tensors is a no-op") 2022-11-23T02:49:35.4908288Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjgurqikc 2022-11-23T02:49:35.4908544Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjgurqikc/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4908932Z /opt/conda/lib/python3.8/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op 2022-11-23T02:49:35.4909243Z warnings.warn("Initializing zero-element tensors is a no-op") 2022-11-23T02:49:35.4909482Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn9cd1agi 2022-11-23T02:49:35.4909733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn9cd1agi/_remote_module_non_scriptable.py 2022-11-23T02:49:35.4909814Z ok (5.421s) 2022-11-23T02:49:35.4909821Z 2022-11-23T02:49:35.4910089Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4910190Z Ran 1 test in 5.422s 2022-11-23T02:49:35.4910197Z 2022-11-23T02:49:35.4910282Z OK 2022-11-23T02:49:35.4910288Z 2022-11-23T02:49:35.4910403Z Generating XML reports... 2022-11-23T02:49:35.4910848Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023707.xml 2022-11-23T02:49:35.4911212Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4911595Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4911764Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4912148Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4912328Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4912334Z 2022-11-23T02:49:35.4912440Z Running tests... 2022-11-23T02:49:35.4912711Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4913014Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76796 2022-11-23T02:49:35.4913226Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76797 2022-11-23T02:49:35.4913490Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4913864Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4914030Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4914414Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4914596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4914822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4915223Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4915580Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4915749Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4916137Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4916319Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4916548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4916943Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4917159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4917376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4917608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:49:35.4917913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:49:35.4918313Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.4918712Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.4918811Z ok (5.027s) 2022-11-23T02:49:35.4918817Z 2022-11-23T02:49:35.4919083Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4919187Z Ran 1 test in 5.028s 2022-11-23T02:49:35.4919193Z 2022-11-23T02:49:35.4919281Z OK 2022-11-23T02:49:35.4919287Z 2022-11-23T02:49:35.4919404Z Generating XML reports... 2022-11-23T02:49:35.4919846Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023717.xml 2022-11-23T02:49:35.4920214Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4920602Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4920771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4921158Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4921338Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4921344Z 2022-11-23T02:49:35.4921432Z Running tests... 2022-11-23T02:49:35.4921701Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4921998Z test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77009 2022-11-23T02:49:35.4922209Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77010 2022-11-23T02:49:35.4922467Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4922844Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4923011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4923398Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4923582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4923808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4924209Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4924581Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4924757Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4925141Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4925328Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4925563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4925957Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4926177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4926394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4926618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:49:35.4927071Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.4927296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:49:35.4927749Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.4927832Z ok (5.035s) 2022-11-23T02:49:35.4927849Z 2022-11-23T02:49:35.4928104Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4928207Z Ran 1 test in 5.036s 2022-11-23T02:49:35.4928213Z 2022-11-23T02:49:35.4928301Z OK 2022-11-23T02:49:35.4928307Z 2022-11-23T02:49:35.4928427Z Generating XML reports... 2022-11-23T02:49:35.4928873Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023727.xml 2022-11-23T02:49:35.4929187Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4929622Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4929801Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4930192Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4930372Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4930378Z 2022-11-23T02:49:35.4930482Z Running tests... 2022-11-23T02:49:35.4930752Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4931672Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78767 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.596s) 2022-11-23T02:49:35.4931682Z 2022-11-23T02:49:35.4931947Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4932048Z Ran 1 test in 0.597s 2022-11-23T02:49:35.4932054Z 2022-11-23T02:49:35.4932156Z OK (skipped=1) 2022-11-23T02:49:35.4932162Z 2022-11-23T02:49:35.4932283Z Generating XML reports... 2022-11-23T02:49:35.4932728Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023736.xml 2022-11-23T02:49:35.4933047Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4933422Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4933592Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4933984Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4934169Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4934176Z 2022-11-23T02:49:35.4934280Z Running tests... 2022-11-23T02:49:35.4934532Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4934982Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.002s) 2022-11-23T02:49:35.4935004Z 2022-11-23T02:49:35.4935256Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4935358Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4935364Z 2022-11-23T02:49:35.4935463Z OK (skipped=1) 2022-11-23T02:49:35.4935469Z 2022-11-23T02:49:35.4935588Z Generating XML reports... 2022-11-23T02:49:35.4936032Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023741.xml 2022-11-23T02:49:35.4936421Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4936797Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4936964Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4937350Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4937533Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4937540Z 2022-11-23T02:49:35.4937644Z Running tests... 2022-11-23T02:49:35.4937912Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4938231Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77354 2022-11-23T02:49:35.4938485Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77355 2022-11-23T02:49:35.4938752Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4939130Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4939299Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4939688Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4939868Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4940094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4940467Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4940619Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4941008Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4941193Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4941418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4941817Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4942212Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4942432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4942649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4942746Z ok (5.115s) 2022-11-23T02:49:35.4942753Z 2022-11-23T02:49:35.4943030Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4943135Z Ran 1 test in 5.115s 2022-11-23T02:49:35.4943142Z 2022-11-23T02:49:35.4943223Z OK 2022-11-23T02:49:35.4943229Z 2022-11-23T02:49:35.4943347Z Generating XML reports... 2022-11-23T02:49:35.4943791Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023745.xml 2022-11-23T02:49:35.4944106Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4944480Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4944645Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4945030Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4945210Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4945274Z 2022-11-23T02:49:35.4945383Z Running tests... 2022-11-23T02:49:35.4945652Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4945944Z test_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77561 2022-11-23T02:49:35.4946138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77562 2022-11-23T02:49:35.4946399Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4946772Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4946939Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4947322Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4947554Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4947791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4948192Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4948565Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4948734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4949117Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4949296Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4949523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4949926Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4950143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4950486Z STAGE:2022-11-23 02:37:57 77562:77562 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4950705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4951042Z STAGE:2022-11-23 02:37:57 77561:77561 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4951386Z STAGE:2022-11-23 02:37:57 77562:77562 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4951726Z STAGE:2022-11-23 02:37:57 77561:77561 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4952082Z STAGE:2022-11-23 02:37:57 77562:77562 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4952447Z STAGE:2022-11-23 02:37:57 77561:77561 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4952785Z STAGE:2022-11-23 02:37:57 77562:77562 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4953116Z STAGE:2022-11-23 02:37:57 77561:77561 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4953440Z STAGE:2022-11-23 02:37:57 77561:77561 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4953794Z STAGE:2022-11-23 02:37:57 77561:77561 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4954136Z STAGE:2022-11-23 02:37:57 77562:77562 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4954490Z STAGE:2022-11-23 02:37:57 77562:77562 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4954587Z ok (5.143s) 2022-11-23T02:49:35.4954593Z 2022-11-23T02:49:35.4954866Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4955022Z Ran 1 test in 5.144s 2022-11-23T02:49:35.4955029Z 2022-11-23T02:49:35.4955119Z OK 2022-11-23T02:49:35.4955124Z 2022-11-23T02:49:35.4955245Z Generating XML reports... 2022-11-23T02:49:35.4955692Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023754.xml 2022-11-23T02:49:35.4956009Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4956387Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4956555Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4956945Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4957126Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4957136Z 2022-11-23T02:49:35.4957300Z Running tests... 2022-11-23T02:49:35.4957574Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4957873Z test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77774 2022-11-23T02:49:35.4958082Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77775 2022-11-23T02:49:35.4958342Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4958720Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4958889Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4959257Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4959449Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4959674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4960050Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4960219Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4960603Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4960782Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4961007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4961409Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4961810Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4962033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4962251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4962347Z ok (5.423s) 2022-11-23T02:49:35.4962353Z 2022-11-23T02:49:35.4962627Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4962731Z Ran 1 test in 5.423s 2022-11-23T02:49:35.4962737Z 2022-11-23T02:49:35.4962825Z OK 2022-11-23T02:49:35.4962830Z 2022-11-23T02:49:35.4962951Z Generating XML reports... 2022-11-23T02:49:35.4963396Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023803.xml 2022-11-23T02:49:35.4963713Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4964093Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4964318Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4964710Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4964875Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4964897Z 2022-11-23T02:49:35.4964984Z Running tests... 2022-11-23T02:49:35.4965254Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4965495Z test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s) 2022-11-23T02:49:35.4965501Z 2022-11-23T02:49:35.4965771Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4965874Z Ran 1 test in 0.002s 2022-11-23T02:49:35.4965880Z 2022-11-23T02:49:35.4965979Z OK (skipped=1) 2022-11-23T02:49:35.4965990Z 2022-11-23T02:49:35.4966157Z Generating XML reports... 2022-11-23T02:49:35.4966606Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023813.xml 2022-11-23T02:49:35.4966925Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4967303Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4967473Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4967910Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4968090Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4968097Z 2022-11-23T02:49:35.4968198Z Running tests... 2022-11-23T02:49:35.4968468Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4968783Z test_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78047 2022-11-23T02:49:35.4968992Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78048 2022-11-23T02:49:35.4969248Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4969622Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4969789Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4970175Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4970357Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4970567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4970946Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4971114Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4971497Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4971678Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4971900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4972303Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4972704Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4972928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4973214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4973443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:49:35.4973674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:49:35.4974074Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.4974468Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.4974804Z STAGE:2022-11-23 02:38:20 78047:78047 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4975138Z STAGE:2022-11-23 02:38:20 78048:78048 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4975529Z STAGE:2022-11-23 02:38:20 78047:78047 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4975896Z STAGE:2022-11-23 02:38:20 78047:78047 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4976239Z STAGE:2022-11-23 02:38:20 78048:78048 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4976595Z STAGE:2022-11-23 02:38:20 78048:78048 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4976927Z STAGE:2022-11-23 02:38:20 78048:78048 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4977260Z STAGE:2022-11-23 02:38:20 78047:78047 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.4977599Z STAGE:2022-11-23 02:38:20 78047:78047 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4977955Z STAGE:2022-11-23 02:38:20 78047:78047 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4978285Z STAGE:2022-11-23 02:38:20 78048:78048 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.4978638Z STAGE:2022-11-23 02:38:20 78048:78048 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.4978734Z ok (5.120s) 2022-11-23T02:49:35.4978740Z 2022-11-23T02:49:35.4979005Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4979110Z Ran 1 test in 5.121s 2022-11-23T02:49:35.4979116Z 2022-11-23T02:49:35.4979205Z OK 2022-11-23T02:49:35.4979210Z 2022-11-23T02:49:35.4979330Z Generating XML reports... 2022-11-23T02:49:35.4979774Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023817.xml 2022-11-23T02:49:35.4980090Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4980467Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4980641Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4981029Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4981208Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4981214Z 2022-11-23T02:49:35.4981318Z Running tests... 2022-11-23T02:49:35.4981588Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4981887Z test_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78266 2022-11-23T02:49:35.4982095Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78267 2022-11-23T02:49:35.4982357Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4982741Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4982967Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4983356Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4983537Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4983750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4984123Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4984291Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4984675Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4984853Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4985144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4985550Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4985942Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4986158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4986377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4986533Z skip: Skipped due to small world size. (4.915s) 2022-11-23T02:49:35.4986539Z 2022-11-23T02:49:35.4986807Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4986910Z Ran 1 test in 4.916s 2022-11-23T02:49:35.4986917Z 2022-11-23T02:49:35.4987018Z OK (skipped=1) 2022-11-23T02:49:35.4987029Z 2022-11-23T02:49:35.4987151Z Generating XML reports... 2022-11-23T02:49:35.4987598Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023827.xml 2022-11-23T02:49:35.4987914Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4988292Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4988461Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4988848Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4989029Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4989036Z 2022-11-23T02:49:35.4989137Z Running tests... 2022-11-23T02:49:35.4989391Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4989695Z test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78473 2022-11-23T02:49:35.4989905Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78474 2022-11-23T02:49:35.4990161Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.4990537Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4990706Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4991087Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4991265Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4991487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.4991923Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4992085Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4992463Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4992642Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4992861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.4993258Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4993647Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.4993905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.4994123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.4994216Z ok (5.420s) 2022-11-23T02:49:35.4994222Z 2022-11-23T02:49:35.4994487Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4994584Z Ran 1 test in 5.421s 2022-11-23T02:49:35.4994591Z 2022-11-23T02:49:35.4994674Z OK 2022-11-23T02:49:35.4994680Z 2022-11-23T02:49:35.4994793Z Generating XML reports... 2022-11-23T02:49:35.4995220Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023836.xml 2022-11-23T02:49:35.4995529Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4995896Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4996059Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.4996445Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.4996621Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.4996627Z 2022-11-23T02:49:35.4996727Z Running tests... 2022-11-23T02:49:35.4996992Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4997894Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.586s) 2022-11-23T02:49:35.4997901Z 2022-11-23T02:49:35.4998162Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.4998260Z Ran 1 test in 0.586s 2022-11-23T02:49:35.4998272Z 2022-11-23T02:49:35.4998370Z OK (skipped=1) 2022-11-23T02:49:35.4998376Z 2022-11-23T02:49:35.4998488Z Generating XML reports... 2022-11-23T02:49:35.4998924Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023845.xml 2022-11-23T02:49:35.4999234Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.4999601Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.4999761Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5000143Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5000319Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5000325Z 2022-11-23T02:49:35.5000423Z Running tests... 2022-11-23T02:49:35.5000751Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5001046Z test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78746 2022-11-23T02:49:35.5001253Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78747 2022-11-23T02:49:35.5001505Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5001873Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5002025Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5002406Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5002582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5002856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5003229Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5003396Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5003775Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5003952Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5004178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5004575Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5004965Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5005188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5005397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5005620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:49:35.5005845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:49:35.5006237Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.5006621Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.5006717Z ok (5.029s) 2022-11-23T02:49:35.5006723Z 2022-11-23T02:49:35.5006988Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5007086Z Ran 1 test in 5.030s 2022-11-23T02:49:35.5007095Z 2022-11-23T02:49:35.5007183Z OK 2022-11-23T02:49:35.5007189Z 2022-11-23T02:49:35.5007300Z Generating XML reports... 2022-11-23T02:49:35.5007861Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023851.xml 2022-11-23T02:49:35.5008167Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5008537Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5008700Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5009077Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5009252Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5009258Z 2022-11-23T02:49:35.5009356Z Running tests... 2022-11-23T02:49:35.5009622Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5009980Z test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78959 2022-11-23T02:49:35.5010186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78960 2022-11-23T02:49:35.5010437Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5010812Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5010975Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5011356Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5011532Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5011799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5012179Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5012345Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5012725Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5012903Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5013121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5013514Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5013908Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5014128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5014328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5014417Z ok (5.069s) 2022-11-23T02:49:35.5014423Z 2022-11-23T02:49:35.5014687Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5014785Z Ran 1 test in 5.069s 2022-11-23T02:49:35.5014791Z 2022-11-23T02:49:35.5014877Z OK 2022-11-23T02:49:35.5014883Z 2022-11-23T02:49:35.5014994Z Generating XML reports... 2022-11-23T02:49:35.5015434Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023900.xml 2022-11-23T02:49:35.5015746Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5016116Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5016284Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5016663Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5016839Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5016846Z 2022-11-23T02:49:35.5016942Z Running tests... 2022-11-23T02:49:35.5017205Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5017490Z test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79166 2022-11-23T02:49:35.5017694Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79167 2022-11-23T02:49:35.5017944Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5018316Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5018539Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5018919Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5019094Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5019316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5019676Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5019837Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5020216Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5020393Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5020664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5021064Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5021455Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5021665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5021877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5021970Z ok (5.187s) 2022-11-23T02:49:35.5021977Z 2022-11-23T02:49:35.5022242Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5022341Z Ran 1 test in 5.187s 2022-11-23T02:49:35.5022347Z 2022-11-23T02:49:35.5022432Z OK 2022-11-23T02:49:35.5022437Z 2022-11-23T02:49:35.5022550Z Generating XML reports... 2022-11-23T02:49:35.5022996Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023909.xml 2022-11-23T02:49:35.5023307Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5023675Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5023838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5024218Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5024392Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5024398Z 2022-11-23T02:49:35.5024494Z Running tests... 2022-11-23T02:49:35.5024749Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5025055Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79373 2022-11-23T02:49:35.5025264Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79374 2022-11-23T02:49:35.5025516Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5025885Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5026048Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5026426Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5026600Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5026822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5027192Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5027414Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5027796Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5027971Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5028189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5028584Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5028972Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5029186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5029446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5029666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:49:35.5029888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:49:35.5030281Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.5030666Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.5030762Z ok (5.221s) 2022-11-23T02:49:35.5030768Z 2022-11-23T02:49:35.5031034Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5031121Z Ran 1 test in 5.221s 2022-11-23T02:49:35.5031136Z 2022-11-23T02:49:35.5031208Z OK 2022-11-23T02:49:35.5031213Z 2022-11-23T02:49:35.5031327Z Generating XML reports... 2022-11-23T02:49:35.5031767Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023919.xml 2022-11-23T02:49:35.5032078Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5032446Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5032608Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5032989Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5033164Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5033171Z 2022-11-23T02:49:35.5033266Z Running tests... 2022-11-23T02:49:35.5033529Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5033832Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79586 2022-11-23T02:49:35.5034038Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79587 2022-11-23T02:49:35.5034291Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5034661Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5034825Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5035203Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5035383Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5035601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5035998Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5036501Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5036665Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5037047Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5037213Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5037436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5037829Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5038042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5038302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5038529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:49:35.5038744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:49:35.5039133Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.5039522Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.5039612Z ok (5.215s) 2022-11-23T02:49:35.5039619Z 2022-11-23T02:49:35.5039885Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5039982Z Ran 1 test in 5.215s 2022-11-23T02:49:35.5039988Z 2022-11-23T02:49:35.5040070Z OK 2022-11-23T02:49:35.5040076Z 2022-11-23T02:49:35.5040190Z Generating XML reports... 2022-11-23T02:49:35.5040629Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023928.xml 2022-11-23T02:49:35.5040940Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5041310Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5041475Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5041854Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5042029Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5042035Z 2022-11-23T02:49:35.5042132Z Running tests... 2022-11-23T02:49:35.5042399Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5042822Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.003s) 2022-11-23T02:49:35.5042839Z 2022-11-23T02:49:35.5043092Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5043189Z Ran 1 test in 0.003s 2022-11-23T02:49:35.5043195Z 2022-11-23T02:49:35.5043288Z OK (skipped=1) 2022-11-23T02:49:35.5043293Z 2022-11-23T02:49:35.5043406Z Generating XML reports... 2022-11-23T02:49:35.5043845Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023938.xml 2022-11-23T02:49:35.5044159Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5044531Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5044694Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5045076Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5045316Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5045322Z 2022-11-23T02:49:35.5045421Z Running tests... 2022-11-23T02:49:35.5045687Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5045972Z test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79865 2022-11-23T02:49:35.5046180Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79866 2022-11-23T02:49:35.5046429Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5046798Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5046960Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5047389Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5047569Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5047827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5048200Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5048363Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5048732Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5048909Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5049130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5049527Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5049916Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5050133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5050345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5050435Z ok (5.218s) 2022-11-23T02:49:35.5050441Z 2022-11-23T02:49:35.5050707Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5050804Z Ran 1 test in 5.218s 2022-11-23T02:49:35.5050810Z 2022-11-23T02:49:35.5050891Z OK 2022-11-23T02:49:35.5050896Z 2022-11-23T02:49:35.5051008Z Generating XML reports... 2022-11-23T02:49:35.5051449Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023943.xml 2022-11-23T02:49:35.5051764Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5052135Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5052295Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5074756Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5074942Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5074951Z 2022-11-23T02:49:35.5075068Z Running tests... 2022-11-23T02:49:35.5075338Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5075619Z test_isend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80072 2022-11-23T02:49:35.5075823Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80073 2022-11-23T02:49:35.5076237Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5076611Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5076788Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5077179Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5077356Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5077576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5077969Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5078482Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5078654Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5079036Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5079206Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5079424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5079817Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5080023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5080231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5080319Z ok (5.213s) 2022-11-23T02:49:35.5080326Z 2022-11-23T02:49:35.5080589Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5080682Z Ran 1 test in 5.213s 2022-11-23T02:49:35.5080688Z 2022-11-23T02:49:35.5080766Z OK 2022-11-23T02:49:35.5080773Z 2022-11-23T02:49:35.5080883Z Generating XML reports... 2022-11-23T02:49:35.5081318Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023952.xml 2022-11-23T02:49:35.5081625Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5081990Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5082154Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5082534Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5082704Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5082717Z 2022-11-23T02:49:35.5082813Z Running tests... 2022-11-23T02:49:35.5083072Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5083377Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80279 2022-11-23T02:49:35.5083577Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80280 2022-11-23T02:49:35.5083829Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5084193Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5084350Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5084725Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5084965Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5085204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5085618Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5085988Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5086140Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5086513Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5086682Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5086899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5087336Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5087560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5088083Z STAGE:2022-11-23 02:40:04 80280:80280 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5088303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5088637Z STAGE:2022-11-23 02:40:04 80279:80279 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5088974Z STAGE:2022-11-23 02:40:04 80280:80280 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5089329Z STAGE:2022-11-23 02:40:04 80280:80280 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5089667Z STAGE:2022-11-23 02:40:04 80279:80279 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5090029Z STAGE:2022-11-23 02:40:04 80279:80279 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5090124Z ok (5.218s) 2022-11-23T02:49:35.5090130Z 2022-11-23T02:49:35.5090398Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5090497Z Ran 1 test in 5.219s 2022-11-23T02:49:35.5090504Z 2022-11-23T02:49:35.5090587Z OK 2022-11-23T02:49:35.5090593Z 2022-11-23T02:49:35.5090708Z Generating XML reports... 2022-11-23T02:49:35.5091148Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024001.xml 2022-11-23T02:49:35.5091467Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5091839Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5092003Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5092396Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5092562Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5092581Z 2022-11-23T02:49:35.5092669Z Running tests... 2022-11-23T02:49:35.5092934Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5093238Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80492 2022-11-23T02:49:35.5093447Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80493 2022-11-23T02:49:35.5093704Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5094078Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5094318Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5094702Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5094881Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5095103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5095475Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5095638Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5096026Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5096204Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5096429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5096880Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5097278Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5097494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5097825Z STAGE:2022-11-23 02:40:13 80493:80493 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5098038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5098371Z STAGE:2022-11-23 02:40:14 80492:80492 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5098709Z STAGE:2022-11-23 02:40:14 80493:80493 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5099053Z STAGE:2022-11-23 02:40:14 80493:80493 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5099394Z STAGE:2022-11-23 02:40:14 80492:80492 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5099744Z STAGE:2022-11-23 02:40:14 80492:80492 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5099839Z ok (5.013s) 2022-11-23T02:49:35.5099845Z 2022-11-23T02:49:35.5100111Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5100212Z Ran 1 test in 5.014s 2022-11-23T02:49:35.5100218Z 2022-11-23T02:49:35.5100302Z OK 2022-11-23T02:49:35.5100307Z 2022-11-23T02:49:35.5100422Z Generating XML reports... 2022-11-23T02:49:35.5100877Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024011.xml 2022-11-23T02:49:35.5101192Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5101570Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5101739Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5102126Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5102305Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5102311Z 2022-11-23T02:49:35.5102414Z Running tests... 2022-11-23T02:49:35.5102680Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5103140Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.001s) 2022-11-23T02:49:35.5103146Z 2022-11-23T02:49:35.5103409Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5103510Z Ran 1 test in 0.002s 2022-11-23T02:49:35.5103515Z 2022-11-23T02:49:35.5103677Z OK (skipped=1) 2022-11-23T02:49:35.5103682Z 2022-11-23T02:49:35.5103801Z Generating XML reports... 2022-11-23T02:49:35.5104242Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024020.xml 2022-11-23T02:49:35.5104545Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5104918Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5105085Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5105473Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5105651Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5105657Z 2022-11-23T02:49:35.5105758Z Running tests... 2022-11-23T02:49:35.5106073Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5106569Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.001s) 2022-11-23T02:49:35.5106575Z 2022-11-23T02:49:35.5106840Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5106939Z Ran 1 test in 0.002s 2022-11-23T02:49:35.5106944Z 2022-11-23T02:49:35.5107042Z OK (skipped=1) 2022-11-23T02:49:35.5107048Z 2022-11-23T02:49:35.5107163Z Generating XML reports... 2022-11-23T02:49:35.5107607Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024025.xml 2022-11-23T02:49:35.5107919Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5108293Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5108466Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5108851Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5109029Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5109034Z 2022-11-23T02:49:35.5109133Z Running tests... 2022-11-23T02:49:35.5109400Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5109719Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80837 2022-11-23T02:49:35.5109928Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80838 2022-11-23T02:49:35.5110180Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5110556Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5110712Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5111093Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5111272Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5111498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5111871Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5112036Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5112417Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5112595Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5112885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5113284Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5113675Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5113890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5114106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5114253Z skip: Skipped due to small world size. (4.914s) 2022-11-23T02:49:35.5114259Z 2022-11-23T02:49:35.5114526Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5114627Z Ran 1 test in 4.915s 2022-11-23T02:49:35.5114633Z 2022-11-23T02:49:35.5114730Z OK (skipped=1) 2022-11-23T02:49:35.5114739Z 2022-11-23T02:49:35.5114900Z Generating XML reports... 2022-11-23T02:49:35.5115348Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024029.xml 2022-11-23T02:49:35.5115661Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5116034Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5116199Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5116568Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5116747Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5116766Z 2022-11-23T02:49:35.5116855Z Running tests... 2022-11-23T02:49:35.5117120Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5117436Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81044 2022-11-23T02:49:35.5117642Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81045 2022-11-23T02:49:35.5117899Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5118271Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5118435Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5118816Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5118991Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5119216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5119593Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5119757Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5120138Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5120317Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5120540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5120934Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5121325Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5121548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5121815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5122029Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms 2022-11-23T02:49:35.5122122Z ok (7.023s) 2022-11-23T02:49:35.5122128Z 2022-11-23T02:49:35.5122385Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5122486Z Ran 1 test in 7.024s 2022-11-23T02:49:35.5122492Z 2022-11-23T02:49:35.5122575Z OK 2022-11-23T02:49:35.5122581Z 2022-11-23T02:49:35.5122696Z Generating XML reports... 2022-11-23T02:49:35.5123135Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024038.xml 2022-11-23T02:49:35.5123448Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5123865Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5124038Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5124422Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5124601Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5124607Z 2022-11-23T02:49:35.5124708Z Running tests... 2022-11-23T02:49:35.5124975Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5125299Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81251 2022-11-23T02:49:35.5125504Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81252 2022-11-23T02:49:35.5125759Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5126140Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5126303Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5126685Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5126861Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5127088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5127457Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5127625Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5128053Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5128241Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5128464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5128858Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5129251Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5129464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5129679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5129902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:49:35.5130122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:49:35.5130587Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.5130790Z [E ProcessGroupGloo.cpp:137] Rank 0 timed out in monitoredBarrier after 0 ms. 2022-11-23T02:49:35.5130955Z No ranks successfully processed in monitoredBarrier. 2022-11-23T02:49:35.5131347Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.5131556Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms 2022-11-23T02:49:35.5131649Z ok (5.112s) 2022-11-23T02:49:35.5131655Z 2022-11-23T02:49:35.5131922Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5132022Z Ran 1 test in 5.112s 2022-11-23T02:49:35.5132027Z 2022-11-23T02:49:35.5132112Z OK 2022-11-23T02:49:35.5132117Z 2022-11-23T02:49:35.5132233Z Generating XML reports... 2022-11-23T02:49:35.5132733Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024049.xml 2022-11-23T02:49:35.5133059Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5133431Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5133583Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5133967Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5134147Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5134153Z 2022-11-23T02:49:35.5134254Z Running tests... 2022-11-23T02:49:35.5134518Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5134837Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81464 2022-11-23T02:49:35.5135048Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81465 2022-11-23T02:49:35.5135302Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5135674Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5135838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5136220Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5136399Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5136623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5136997Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5137163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5137547Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5137727Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5137958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5138353Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5138745Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5138964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5139177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5139478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:49:35.5139701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:49:35.5140086Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.5140474Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.5140686Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms 2022-11-23T02:49:35.5140780Z ok (5.117s) 2022-11-23T02:49:35.5140785Z 2022-11-23T02:49:35.5141052Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5141152Z Ran 1 test in 5.117s 2022-11-23T02:49:35.5141158Z 2022-11-23T02:49:35.5141241Z OK 2022-11-23T02:49:35.5141247Z 2022-11-23T02:49:35.5141415Z Generating XML reports... 2022-11-23T02:49:35.5141861Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024058.xml 2022-11-23T02:49:35.5142173Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5142544Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5142708Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5143090Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5143267Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5143273Z 2022-11-23T02:49:35.5143373Z Running tests... 2022-11-23T02:49:35.5143639Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5143962Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81677 2022-11-23T02:49:35.5144171Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81678 2022-11-23T02:49:35.5144424Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5144794Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5144958Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5145338Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5145504Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5145729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5146106Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5146270Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5146652Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5146833Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5147056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5147455Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5147848Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5148065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5148336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5148485Z skip: Skipped due to small world size. (5.420s) 2022-11-23T02:49:35.5148491Z 2022-11-23T02:49:35.5148761Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5148862Z Ran 1 test in 5.421s 2022-11-23T02:49:35.5148868Z 2022-11-23T02:49:35.5148964Z OK (skipped=1) 2022-11-23T02:49:35.5148970Z 2022-11-23T02:49:35.5149085Z Generating XML reports... 2022-11-23T02:49:35.5149529Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024108.xml 2022-11-23T02:49:35.5149841Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5150214Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5150423Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5150817Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5150994Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5151000Z 2022-11-23T02:49:35.5151100Z Running tests... 2022-11-23T02:49:35.5151354Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5151759Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s) 2022-11-23T02:49:35.5151765Z 2022-11-23T02:49:35.5152029Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5152129Z Ran 1 test in 0.002s 2022-11-23T02:49:35.5152134Z 2022-11-23T02:49:35.5152230Z OK (skipped=1) 2022-11-23T02:49:35.5152236Z 2022-11-23T02:49:35.5152351Z Generating XML reports... 2022-11-23T02:49:35.5152796Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024117.xml 2022-11-23T02:49:35.5153112Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5153482Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5153644Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5154024Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5154199Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5154205Z 2022-11-23T02:49:35.5154303Z Running tests... 2022-11-23T02:49:35.5154570Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5154982Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s) 2022-11-23T02:49:35.5154991Z 2022-11-23T02:49:35.5155256Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5155356Z Ran 1 test in 0.003s 2022-11-23T02:49:35.5155361Z 2022-11-23T02:49:35.5155458Z OK (skipped=1) 2022-11-23T02:49:35.5155463Z 2022-11-23T02:49:35.5155577Z Generating XML reports... 2022-11-23T02:49:35.5156016Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024121.xml 2022-11-23T02:49:35.5156328Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5156700Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5156865Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5157236Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5157476Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5157482Z 2022-11-23T02:49:35.5157582Z Running tests... 2022-11-23T02:49:35.5157851Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5158262Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s) 2022-11-23T02:49:35.5158268Z 2022-11-23T02:49:35.5158531Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5158632Z Ran 1 test in 0.002s 2022-11-23T02:49:35.5158638Z 2022-11-23T02:49:35.5158734Z OK (skipped=1) 2022-11-23T02:49:35.5158740Z 2022-11-23T02:49:35.5158856Z Generating XML reports... 2022-11-23T02:49:35.5159296Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024126.xml 2022-11-23T02:49:35.5159657Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5160039Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5160207Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5160589Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5160769Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5160775Z 2022-11-23T02:49:35.5160875Z Running tests... 2022-11-23T02:49:35.5161141Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5161540Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s) 2022-11-23T02:49:35.5161546Z 2022-11-23T02:49:35.5161808Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5161910Z Ran 1 test in 0.003s 2022-11-23T02:49:35.5161920Z 2022-11-23T02:49:35.5162017Z OK (skipped=1) 2022-11-23T02:49:35.5162022Z 2022-11-23T02:49:35.5162137Z Generating XML reports... 2022-11-23T02:49:35.5162577Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024131.xml 2022-11-23T02:49:35.5162876Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5163252Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5163416Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5163800Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5163979Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5163985Z 2022-11-23T02:49:35.5164084Z Running tests... 2022-11-23T02:49:35.5164357Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5164645Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.002s) 2022-11-23T02:49:35.5164651Z 2022-11-23T02:49:35.5164915Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5165018Z Ran 1 test in 0.002s 2022-11-23T02:49:35.5165023Z 2022-11-23T02:49:35.5165122Z OK (skipped=1) 2022-11-23T02:49:35.5165127Z 2022-11-23T02:49:35.5165243Z Generating XML reports... 2022-11-23T02:49:35.5165707Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024135.xml 2022-11-23T02:49:35.5166020Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5166392Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5166617Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5167004Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5167180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5167186Z 2022-11-23T02:49:35.5167285Z Running tests... 2022-11-23T02:49:35.5167554Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5167971Z test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:49:35.5167978Z 2022-11-23T02:49:35.5168253Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5168355Z Ran 1 test in 0.002s 2022-11-23T02:49:35.5168361Z 2022-11-23T02:49:35.5168447Z OK (skipped=1) 2022-11-23T02:49:35.5168465Z 2022-11-23T02:49:35.5168568Z Generating XML reports... 2022-11-23T02:49:35.5169074Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024139.xml 2022-11-23T02:49:35.5169401Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5169773Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5169940Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5170324Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5170499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5170505Z 2022-11-23T02:49:35.5170603Z Running tests... 2022-11-23T02:49:35.5170866Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5171118Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:49:35.5171131Z 2022-11-23T02:49:35.5171393Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5171490Z Ran 1 test in 0.002s 2022-11-23T02:49:35.5171496Z 2022-11-23T02:49:35.5171592Z OK (skipped=1) 2022-11-23T02:49:35.5171597Z 2022-11-23T02:49:35.5171714Z Generating XML reports... 2022-11-23T02:49:35.5172148Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024143.xml 2022-11-23T02:49:35.5172458Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5172826Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5172988Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5173369Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5173551Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5173557Z 2022-11-23T02:49:35.5173656Z Running tests... 2022-11-23T02:49:35.5173910Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5174204Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:49:35.5174221Z 2022-11-23T02:49:35.5174473Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5174576Z Ran 1 test in 0.002s 2022-11-23T02:49:35.5174581Z 2022-11-23T02:49:35.5174677Z OK (skipped=1) 2022-11-23T02:49:35.5174683Z 2022-11-23T02:49:35.5174798Z Generating XML reports... 2022-11-23T02:49:35.5175235Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024147.xml 2022-11-23T02:49:35.5175548Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5175980Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5176143Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5176523Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5176701Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5176707Z 2022-11-23T02:49:35.5176805Z Running tests... 2022-11-23T02:49:35.5177067Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5177406Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82412 2022-11-23T02:49:35.5177612Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82413 2022-11-23T02:49:35.5177917Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5178294Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5178456Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5178835Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5179009Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5179231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5179595Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5179756Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5180132Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5180305Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5180527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5180918Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5181304Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5181515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5181725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5181815Z ok (5.223s) 2022-11-23T02:49:35.5181821Z 2022-11-23T02:49:35.5182090Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5182192Z Ran 1 test in 5.223s 2022-11-23T02:49:35.5182199Z 2022-11-23T02:49:35.5182281Z OK 2022-11-23T02:49:35.5182286Z 2022-11-23T02:49:35.5182398Z Generating XML reports... 2022-11-23T02:49:35.5182832Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024151.xml 2022-11-23T02:49:35.5183140Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5183509Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5183671Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5184049Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5184221Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5184280Z 2022-11-23T02:49:35.5184382Z Running tests... 2022-11-23T02:49:35.5184649Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5184980Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82619 2022-11-23T02:49:35.5185184Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82620 2022-11-23T02:49:35.5185428Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5185796Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5185956Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5186332Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5186560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5186783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5187156Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5187315Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5187693Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5187867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5188087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5188475Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5188869Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5189081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5189293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5189382Z ok (4.911s) 2022-11-23T02:49:35.5189389Z 2022-11-23T02:49:35.5189652Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5189749Z Ran 1 test in 4.912s 2022-11-23T02:49:35.5189755Z 2022-11-23T02:49:35.5189835Z OK 2022-11-23T02:49:35.5189841Z 2022-11-23T02:49:35.5189951Z Generating XML reports... 2022-11-23T02:49:35.5190393Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024200.xml 2022-11-23T02:49:35.5190701Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5191076Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5191229Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5191607Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5191780Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5191786Z 2022-11-23T02:49:35.5191883Z Running tests... 2022-11-23T02:49:35.5192146Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5192402Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:49:35.5192408Z 2022-11-23T02:49:35.5192669Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5192766Z Ran 1 test in 0.002s 2022-11-23T02:49:35.5192772Z 2022-11-23T02:49:35.5192921Z OK (skipped=1) 2022-11-23T02:49:35.5192927Z 2022-11-23T02:49:35.5193039Z Generating XML reports... 2022-11-23T02:49:35.5193478Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024209.xml 2022-11-23T02:49:35.5193791Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5194158Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5194318Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5194700Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5194875Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5194881Z 2022-11-23T02:49:35.5194977Z Running tests... 2022-11-23T02:49:35.5195289Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5195575Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:49:35.5195582Z 2022-11-23T02:49:35.5195847Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5195943Z Ran 1 test in 0.002s 2022-11-23T02:49:35.5195948Z 2022-11-23T02:49:35.5196041Z OK (skipped=1) 2022-11-23T02:49:35.5196046Z 2022-11-23T02:49:35.5196157Z Generating XML reports... 2022-11-23T02:49:35.5196585Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024214.xml 2022-11-23T02:49:35.5196895Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5197263Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5197428Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5197810Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5197986Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5197992Z 2022-11-23T02:49:35.5198087Z Running tests... 2022-11-23T02:49:35.5198348Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5199276Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.587s) 2022-11-23T02:49:35.5199283Z 2022-11-23T02:49:35.5199540Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5199642Z Ran 1 test in 0.587s 2022-11-23T02:49:35.5199648Z 2022-11-23T02:49:35.5199741Z OK (skipped=1) 2022-11-23T02:49:35.5199747Z 2022-11-23T02:49:35.5199858Z Generating XML reports... 2022-11-23T02:49:35.5200293Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024218.xml 2022-11-23T02:49:35.5200604Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5200974Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5201136Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5201514Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5201687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5201693Z 2022-11-23T02:49:35.5201788Z Running tests... 2022-11-23T02:49:35.5202126Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5202444Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83024 2022-11-23T02:49:35.5202648Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83025 2022-11-23T02:49:35.5202898Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5203267Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5203429Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5203798Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5203972Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5204263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5204637Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5204808Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5205187Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5205363Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5205584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5205975Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5206366Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5206579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5206787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5207024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpblin4mnq 2022-11-23T02:49:35.5207272Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpblin4mnq/_remote_module_non_scriptable.py 2022-11-23T02:49:35.5207505Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo16hkpvr 2022-11-23T02:49:35.5207815Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo16hkpvr/_remote_module_non_scriptable.py 2022-11-23T02:49:35.5207905Z ok (7.416s) 2022-11-23T02:49:35.5207911Z 2022-11-23T02:49:35.5208180Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5208277Z Ran 1 test in 7.416s 2022-11-23T02:49:35.5208287Z 2022-11-23T02:49:35.5208370Z OK 2022-11-23T02:49:35.5208376Z 2022-11-23T02:49:35.5208489Z Generating XML reports... 2022-11-23T02:49:35.5208916Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024223.xml 2022-11-23T02:49:35.5209226Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5209596Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5209756Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5210136Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5210311Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5210317Z 2022-11-23T02:49:35.5210411Z Running tests... 2022-11-23T02:49:35.5210677Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5211060Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83241 2022-11-23T02:49:35.5211263Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83242 2022-11-23T02:49:35.5211514Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5211884Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5212046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5212424Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5212596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5212872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5213253Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5213414Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5213790Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5213963Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5214183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5214577Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5214966Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5215184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5215386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5215474Z ok (5.728s) 2022-11-23T02:49:35.5215480Z 2022-11-23T02:49:35.5215742Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5215838Z Ran 1 test in 5.729s 2022-11-23T02:49:35.5215844Z 2022-11-23T02:49:35.5215924Z OK 2022-11-23T02:49:35.5215929Z 2022-11-23T02:49:35.5216042Z Generating XML reports... 2022-11-23T02:49:35.5216476Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024235.xml 2022-11-23T02:49:35.5216784Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5217152Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5217321Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5217699Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5217873Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5217879Z 2022-11-23T02:49:35.5217973Z Running tests... 2022-11-23T02:49:35.5218236Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5218561Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83452 2022-11-23T02:49:35.5218765Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83453 2022-11-23T02:49:35.5219015Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5219384Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5219602Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5219984Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5220157Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5220376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5220735Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5220898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5221277Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5221451Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5221721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5222118Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5222510Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5222723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5222936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5223029Z ok (5.918s) 2022-11-23T02:49:35.5223035Z 2022-11-23T02:49:35.5223300Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5223398Z Ran 1 test in 5.919s 2022-11-23T02:49:35.5223403Z 2022-11-23T02:49:35.5223485Z OK 2022-11-23T02:49:35.5223490Z 2022-11-23T02:49:35.5223611Z Generating XML reports... 2022-11-23T02:49:35.5224047Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024245.xml 2022-11-23T02:49:35.5224357Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5224729Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5224889Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5225268Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5225443Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5225449Z 2022-11-23T02:49:35.5225548Z Running tests... 2022-11-23T02:49:35.5225801Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5226734Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.583s) 2022-11-23T02:49:35.5226745Z 2022-11-23T02:49:35.5227009Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5227107Z Ran 1 test in 0.583s 2022-11-23T02:49:35.5227113Z 2022-11-23T02:49:35.5227197Z OK (skipped=1) 2022-11-23T02:49:35.5227216Z 2022-11-23T02:49:35.5227318Z Generating XML reports... 2022-11-23T02:49:35.5227752Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024255.xml 2022-11-23T02:49:35.5228064Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5228436Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5228661Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5229047Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5229224Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5229230Z 2022-11-23T02:49:35.5229329Z Running tests... 2022-11-23T02:49:35.5229592Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5230539Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.583s) 2022-11-23T02:49:35.5230597Z 2022-11-23T02:49:35.5230868Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5230966Z Ran 1 test in 0.583s 2022-11-23T02:49:35.5230972Z 2022-11-23T02:49:35.5231068Z OK (skipped=1) 2022-11-23T02:49:35.5231073Z 2022-11-23T02:49:35.5231185Z Generating XML reports... 2022-11-23T02:49:35.5231621Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024259.xml 2022-11-23T02:49:35.5231930Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5232296Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5232461Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5232841Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5233032Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5233038Z 2022-11-23T02:49:35.5233134Z Running tests... 2022-11-23T02:49:35.5233403Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5234439Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75052 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.589s) 2022-11-23T02:49:35.5234446Z 2022-11-23T02:49:35.5234705Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5234809Z Ran 1 test in 0.589s 2022-11-23T02:49:35.5234815Z 2022-11-23T02:49:35.5234908Z OK (skipped=1) 2022-11-23T02:49:35.5234913Z 2022-11-23T02:49:35.5235031Z Generating XML reports... 2022-11-23T02:49:35.5235471Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024305.xml 2022-11-23T02:49:35.5235771Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5236140Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5236301Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5236681Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5236858Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5236864Z 2022-11-23T02:49:35.5236966Z Running tests... 2022-11-23T02:49:35.5237228Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5238216Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75139 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.591s) 2022-11-23T02:49:35.5238283Z 2022-11-23T02:49:35.5238549Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5238648Z Ran 1 test in 0.592s 2022-11-23T02:49:35.5238653Z 2022-11-23T02:49:35.5238750Z OK (skipped=1) 2022-11-23T02:49:35.5238755Z 2022-11-23T02:49:35.5238868Z Generating XML reports... 2022-11-23T02:49:35.5239305Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024310.xml 2022-11-23T02:49:35.5239617Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5240038Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5240202Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5240585Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5240763Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5240769Z 2022-11-23T02:49:35.5240866Z Running tests... 2022-11-23T02:49:35.5241131Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5242067Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84886 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.640s) 2022-11-23T02:49:35.5242077Z 2022-11-23T02:49:35.5242345Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5242444Z Ran 1 test in 0.641s 2022-11-23T02:49:35.5242449Z 2022-11-23T02:49:35.5242547Z OK (skipped=1) 2022-11-23T02:49:35.5242553Z 2022-11-23T02:49:35.5242666Z Generating XML reports... 2022-11-23T02:49:35.5243106Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024315.xml 2022-11-23T02:49:35.5243418Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5243789Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5243952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5244320Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5244506Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5244512Z 2022-11-23T02:49:35.5244611Z Running tests... 2022-11-23T02:49:35.5244874Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5245175Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83993 2022-11-23T02:49:35.5245381Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83994 2022-11-23T02:49:35.5245634Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5246003Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5246165Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5246543Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5246774Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5246998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5247391Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5247812Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5247972Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5248355Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5248528Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5248805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5249204Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5249414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5249620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5249841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:49:35.5250058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:49:35.5250441Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.5250769Z STAGE:2022-11-23 02:43:23 83993:83993 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5251168Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.5251496Z STAGE:2022-11-23 02:43:23 83994:83994 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5251828Z STAGE:2022-11-23 02:43:23 83994:83994 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5252177Z STAGE:2022-11-23 02:43:23 83994:83994 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5252509Z STAGE:2022-11-23 02:43:23 83993:83993 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5252859Z STAGE:2022-11-23 02:43:23 83993:83993 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5253184Z STAGE:2022-11-23 02:43:23 83994:83994 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5253505Z STAGE:2022-11-23 02:43:23 83993:83993 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5253844Z STAGE:2022-11-23 02:43:23 83994:83994 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5254194Z STAGE:2022-11-23 02:43:23 83994:83994 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5254526Z STAGE:2022-11-23 02:43:23 83993:83993 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5254871Z STAGE:2022-11-23 02:43:23 83993:83993 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5254959Z ok (5.619s) 2022-11-23T02:49:35.5254965Z 2022-11-23T02:49:35.5255230Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5255326Z Ran 1 test in 5.620s 2022-11-23T02:49:35.5255331Z 2022-11-23T02:49:35.5255412Z OK 2022-11-23T02:49:35.5255417Z 2022-11-23T02:49:35.5255530Z Generating XML reports... 2022-11-23T02:49:35.5255969Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024320.xml 2022-11-23T02:49:35.5256393Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5256767Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5256928Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5257301Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5257475Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5257489Z 2022-11-23T02:49:35.5257577Z Running tests... 2022-11-23T02:49:35.5257840Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5258142Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84212 2022-11-23T02:49:35.5258399Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84213 2022-11-23T02:49:35.5258655Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5259028Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5259187Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5259566Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5259740Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5259960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5260355Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5260729Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5260890Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5261269Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5261443Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5261661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5262052Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5262262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5262475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5262700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:49:35.5262921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:49:35.5263309Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.5263690Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.5264019Z STAGE:2022-11-23 02:43:33 84212:84212 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5264343Z STAGE:2022-11-23 02:43:33 84213:84213 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5264676Z STAGE:2022-11-23 02:43:33 84212:84212 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5265027Z STAGE:2022-11-23 02:43:33 84212:84212 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5265416Z STAGE:2022-11-23 02:43:33 84213:84213 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5265763Z STAGE:2022-11-23 02:43:33 84213:84213 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5266089Z STAGE:2022-11-23 02:43:33 84212:84212 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5266412Z STAGE:2022-11-23 02:43:33 84213:84213 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5266745Z STAGE:2022-11-23 02:43:33 84212:84212 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5267089Z STAGE:2022-11-23 02:43:33 84212:84212 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5267419Z STAGE:2022-11-23 02:43:33 84213:84213 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5267830Z STAGE:2022-11-23 02:43:33 84213:84213 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5267924Z ok (5.015s) 2022-11-23T02:49:35.5267930Z 2022-11-23T02:49:35.5268198Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5268294Z Ran 1 test in 5.015s 2022-11-23T02:49:35.5268300Z 2022-11-23T02:49:35.5268379Z OK 2022-11-23T02:49:35.5268385Z 2022-11-23T02:49:35.5268497Z Generating XML reports... 2022-11-23T02:49:35.5268935Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024330.xml 2022-11-23T02:49:35.5269244Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5269614Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5269777Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5270162Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5270330Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5270344Z 2022-11-23T02:49:35.5270432Z Running tests... 2022-11-23T02:49:35.5270694Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5271006Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84431 2022-11-23T02:49:35.5271211Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84432 2022-11-23T02:49:35.5271462Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5271832Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5271991Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5272376Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5272550Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5272770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5273138Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5273297Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5273675Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5273848Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5274067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5274522Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5274909Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5275120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5275332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5275549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:49:35.5275765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:49:35.5276157Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.5276582Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.5276918Z STAGE:2022-11-23 02:43:42 84431:84431 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5277246Z STAGE:2022-11-23 02:43:42 84432:84432 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5277577Z STAGE:2022-11-23 02:43:42 84431:84431 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5277924Z STAGE:2022-11-23 02:43:42 84431:84431 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5278257Z STAGE:2022-11-23 02:43:42 84432:84432 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5278601Z STAGE:2022-11-23 02:43:42 84432:84432 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5278929Z STAGE:2022-11-23 02:43:42 84431:84431 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5279257Z STAGE:2022-11-23 02:43:42 84432:84432 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5279587Z STAGE:2022-11-23 02:43:42 84431:84431 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5279932Z STAGE:2022-11-23 02:43:42 84431:84431 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5280262Z STAGE:2022-11-23 02:43:42 84432:84432 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5280608Z STAGE:2022-11-23 02:43:42 84432:84432 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5280698Z ok (5.121s) 2022-11-23T02:49:35.5280704Z 2022-11-23T02:49:35.5280967Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5281062Z Ran 1 test in 5.121s 2022-11-23T02:49:35.5281068Z 2022-11-23T02:49:35.5281150Z OK 2022-11-23T02:49:35.5281156Z 2022-11-23T02:49:35.5281266Z Generating XML reports... 2022-11-23T02:49:35.5281713Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024340.xml 2022-11-23T02:49:35.5282023Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5282396Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5282558Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5282938Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5283104Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5283119Z 2022-11-23T02:49:35.5283207Z Running tests... 2022-11-23T02:49:35.5283473Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5283774Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84650 2022-11-23T02:49:35.5284032Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84651 2022-11-23T02:49:35.5284285Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5284656Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5284817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5285197Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5285368Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5285587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5286002Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5286172Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5286554Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5286727Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5286947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5287338Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5287857Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5288072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5288289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5288509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:49:35.5288728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:49:35.5289128Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.5289518Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.5289837Z STAGE:2022-11-23 02:43:52 84650:84650 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5290160Z STAGE:2022-11-23 02:43:52 84651:84651 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5290493Z STAGE:2022-11-23 02:43:52 84650:84650 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5290850Z STAGE:2022-11-23 02:43:52 84650:84650 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5291180Z STAGE:2022-11-23 02:43:52 84651:84651 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5291525Z STAGE:2022-11-23 02:43:52 84651:84651 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5291850Z STAGE:2022-11-23 02:43:52 84650:84650 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5292172Z STAGE:2022-11-23 02:43:52 84651:84651 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5292500Z STAGE:2022-11-23 02:43:52 84650:84650 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5292841Z STAGE:2022-11-23 02:43:52 84650:84650 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5293167Z STAGE:2022-11-23 02:43:52 84651:84651 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5293589Z STAGE:2022-11-23 02:43:52 84651:84651 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5293679Z ok (5.114s) 2022-11-23T02:49:35.5293685Z 2022-11-23T02:49:35.5293948Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5294047Z Ran 1 test in 5.115s 2022-11-23T02:49:35.5294054Z 2022-11-23T02:49:35.5294134Z OK 2022-11-23T02:49:35.5294140Z 2022-11-23T02:49:35.5294253Z Generating XML reports... 2022-11-23T02:49:35.5294695Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024349.xml 2022-11-23T02:49:35.5295006Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5295377Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5295541Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5295978Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5296145Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5296161Z 2022-11-23T02:49:35.5296249Z Running tests... 2022-11-23T02:49:35.5296524Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5296823Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84869 2022-11-23T02:49:35.5297028Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84870 2022-11-23T02:49:35.5297284Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5297654Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5297823Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5298206Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5298378Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5298607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5298976Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5299137Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5299516Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5299692Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5299917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5300317Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5300707Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5300920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5301135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5301279Z skip: Skipped due to small world size. (4.918s) 2022-11-23T02:49:35.5301285Z 2022-11-23T02:49:35.5301553Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5301651Z Ran 1 test in 4.918s 2022-11-23T02:49:35.5301657Z 2022-11-23T02:49:35.5301742Z OK (skipped=1) 2022-11-23T02:49:35.5301758Z 2022-11-23T02:49:35.5301861Z Generating XML reports... 2022-11-23T02:49:35.5302371Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024358.xml 2022-11-23T02:49:35.5302684Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5303056Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5303220Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5303603Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5303776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5303781Z 2022-11-23T02:49:35.5303879Z Running tests... 2022-11-23T02:49:35.5304147Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5304491Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85076 2022-11-23T02:49:35.5304705Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85077 2022-11-23T02:49:35.5304956Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5305330Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5305495Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5305880Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5306059Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5306285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5306658Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5306829Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5307215Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5307392Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5307615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5307999Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5308392Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5308608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5308827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5308979Z skip: Skipped due to small world size. (5.112s) 2022-11-23T02:49:35.5308985Z 2022-11-23T02:49:35.5309257Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5309357Z Ran 1 test in 5.112s 2022-11-23T02:49:35.5309363Z 2022-11-23T02:49:35.5309462Z OK (skipped=1) 2022-11-23T02:49:35.5309468Z 2022-11-23T02:49:35.5309582Z Generating XML reports... 2022-11-23T02:49:35.5310026Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024407.xml 2022-11-23T02:49:35.5310340Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5310716Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5310879Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5311329Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5311508Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5311514Z 2022-11-23T02:49:35.5311618Z Running tests... 2022-11-23T02:49:35.5311888Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5312195Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85283 2022-11-23T02:49:35.5312402Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85284 2022-11-23T02:49:35.5312660Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5313037Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5313250Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5313641Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5313807Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5314029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5314426Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5314798Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5314963Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5315350Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5315536Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5315764Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5316161Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5316377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5316592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5316743Z skip: Skipped due to small world size. (4.923s) 2022-11-23T02:49:35.5316749Z 2022-11-23T02:49:35.5317018Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5317121Z Ran 1 test in 4.924s 2022-11-23T02:49:35.5317127Z 2022-11-23T02:49:35.5317225Z OK (skipped=1) 2022-11-23T02:49:35.5317231Z 2022-11-23T02:49:35.5317345Z Generating XML reports... 2022-11-23T02:49:35.5317799Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024417.xml 2022-11-23T02:49:35.5318113Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5318489Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5318651Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5319033Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5319210Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5319216Z 2022-11-23T02:49:35.5319303Z Running tests... 2022-11-23T02:49:35.5319568Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5319866Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85490 2022-11-23T02:49:35.5320130Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85491 2022-11-23T02:49:35.5320385Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5320757Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5320923Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5321306Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5321480Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5321700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5322115Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5322321Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5322704Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5322883Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5323109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5323509Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5323896Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5324105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5324322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5324469Z skip: Skipped due to small world size. (4.914s) 2022-11-23T02:49:35.5324475Z 2022-11-23T02:49:35.5324737Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5324834Z Ran 1 test in 4.914s 2022-11-23T02:49:35.5324840Z 2022-11-23T02:49:35.5324933Z OK (skipped=1) 2022-11-23T02:49:35.5324939Z 2022-11-23T02:49:35.5325042Z Generating XML reports... 2022-11-23T02:49:35.5325479Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024426.xml 2022-11-23T02:49:35.5325788Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5326162Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5326322Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5326707Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5326882Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5326888Z 2022-11-23T02:49:35.5326984Z Running tests... 2022-11-23T02:49:35.5327250Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5327537Z test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85697 2022-11-23T02:49:35.5327801Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85698 2022-11-23T02:49:35.5328054Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5328424Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5328588Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5329039Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5329214Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5329433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5329824Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5330192Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5330351Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5330732Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5330907Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5331253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5331640Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5331851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5332180Z STAGE:2022-11-23 02:44:38 85698:85698 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5332391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5332718Z STAGE:2022-11-23 02:44:38 85697:85697 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5333051Z STAGE:2022-11-23 02:44:38 85697:85697 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5333406Z STAGE:2022-11-23 02:44:38 85697:85697 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5333742Z STAGE:2022-11-23 02:44:38 85698:85698 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5334089Z STAGE:2022-11-23 02:44:38 85698:85698 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5334412Z STAGE:2022-11-23 02:44:38 85698:85698 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5334733Z STAGE:2022-11-23 02:44:38 85697:85697 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5335065Z STAGE:2022-11-23 02:44:38 85697:85697 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5335414Z STAGE:2022-11-23 02:44:38 85697:85697 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5335745Z STAGE:2022-11-23 02:44:38 85698:85698 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5336094Z STAGE:2022-11-23 02:44:38 85698:85698 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5336187Z ok (5.141s) 2022-11-23T02:49:35.5336193Z 2022-11-23T02:49:35.5336459Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5336554Z Ran 1 test in 5.141s 2022-11-23T02:49:35.5336560Z 2022-11-23T02:49:35.5336640Z OK 2022-11-23T02:49:35.5336646Z 2022-11-23T02:49:35.5336764Z Generating XML reports... 2022-11-23T02:49:35.5337201Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024435.xml 2022-11-23T02:49:35.5337510Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5337880Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5338032Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5338415Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5338658Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5338664Z 2022-11-23T02:49:35.5338760Z Running tests... 2022-11-23T02:49:35.5339030Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5339315Z test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85910 2022-11-23T02:49:35.5339519Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85911 2022-11-23T02:49:35.5339767Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5340137Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5340298Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5340726Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5340904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5341125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5341498Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5341657Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5342035Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5342211Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5342433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5342833Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5343220Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5343431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5343760Z STAGE:2022-11-23 02:44:47 85911:85911 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5343964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5344290Z STAGE:2022-11-23 02:44:47 85910:85910 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5344840Z STAGE:2022-11-23 02:44:47 85911:85911 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:44:47 85910:85910 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5344859Z 2022-11-23T02:49:35.5345433Z STAGE:2022-11-23 02:44:47 85910:85910 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 02:44:47 85911:85911 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5345447Z 2022-11-23T02:49:35.5345775Z STAGE:2022-11-23 02:44:47 85911:85911 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5346090Z STAGE:2022-11-23 02:44:47 85910:85910 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5346422Z STAGE:2022-11-23 02:44:47 85911:85911 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5346770Z STAGE:2022-11-23 02:44:47 85911:85911 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5347104Z STAGE:2022-11-23 02:44:47 85910:85910 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5347454Z STAGE:2022-11-23 02:44:47 85910:85910 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5347601Z ok (5.318s) 2022-11-23T02:49:35.5347607Z 2022-11-23T02:49:35.5347872Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5347971Z Ran 1 test in 5.319s 2022-11-23T02:49:35.5347977Z 2022-11-23T02:49:35.5348056Z OK 2022-11-23T02:49:35.5348062Z 2022-11-23T02:49:35.5348173Z Generating XML reports... 2022-11-23T02:49:35.5348613Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024444.xml 2022-11-23T02:49:35.5348922Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5349295Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5349457Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5349884Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5350065Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5350071Z 2022-11-23T02:49:35.5350167Z Running tests... 2022-11-23T02:49:35.5350437Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5350695Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s) 2022-11-23T02:49:35.5350701Z 2022-11-23T02:49:35.5350963Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5351059Z Ran 1 test in 0.002s 2022-11-23T02:49:35.5351065Z 2022-11-23T02:49:35.5351156Z OK (skipped=1) 2022-11-23T02:49:35.5351162Z 2022-11-23T02:49:35.5351264Z Generating XML reports... 2022-11-23T02:49:35.5351702Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024454.xml 2022-11-23T02:49:35.5352021Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5352395Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5352555Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5352938Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5353113Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5353120Z 2022-11-23T02:49:35.5353216Z Running tests... 2022-11-23T02:49:35.5353479Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5353774Z test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86189 2022-11-23T02:49:35.5353976Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86190 2022-11-23T02:49:35.5354230Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5354604Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5354765Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5355148Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5355321Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5355542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5355933Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5356305Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5356520Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5356904Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5357078Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5357302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5357683Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5357895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5358220Z STAGE:2022-11-23 02:45:01 86190:86190 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5358428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5358804Z STAGE:2022-11-23 02:45:01 86189:86189 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5359143Z STAGE:2022-11-23 02:45:01 86190:86190 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5359490Z STAGE:2022-11-23 02:45:01 86190:86190 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5359822Z STAGE:2022-11-23 02:45:01 86189:86189 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5360170Z STAGE:2022-11-23 02:45:01 86189:86189 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5360494Z STAGE:2022-11-23 02:45:01 86190:86190 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5360815Z STAGE:2022-11-23 02:45:01 86189:86189 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5361150Z STAGE:2022-11-23 02:45:01 86190:86190 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5361500Z STAGE:2022-11-23 02:45:01 86190:86190 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5361832Z STAGE:2022-11-23 02:45:01 86189:86189 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5362176Z STAGE:2022-11-23 02:45:01 86189:86189 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5362264Z ok (5.019s) 2022-11-23T02:49:35.5362270Z 2022-11-23T02:49:35.5362537Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5362634Z Ran 1 test in 5.019s 2022-11-23T02:49:35.5362640Z 2022-11-23T02:49:35.5362720Z OK 2022-11-23T02:49:35.5362726Z 2022-11-23T02:49:35.5362837Z Generating XML reports... 2022-11-23T02:49:35.5363278Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024458.xml 2022-11-23T02:49:35.5363593Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5363969Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5364122Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5364504Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5364680Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5364686Z 2022-11-23T02:49:35.5364783Z Running tests... 2022-11-23T02:49:35.5365047Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5365320Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce_scatter_tensor (0.002s) 2022-11-23T02:49:35.5365326Z 2022-11-23T02:49:35.5365590Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5365754Z Ran 1 test in 0.002s 2022-11-23T02:49:35.5365759Z 2022-11-23T02:49:35.5365855Z OK (skipped=1) 2022-11-23T02:49:35.5365861Z 2022-11-23T02:49:35.5365972Z Generating XML reports... 2022-11-23T02:49:35.5366412Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024508.xml 2022-11-23T02:49:35.5366722Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5367092Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5367256Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5367637Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5367860Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5367866Z 2022-11-23T02:49:35.5367963Z Running tests... 2022-11-23T02:49:35.5368289Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5368545Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports reduce_scatter_v (0.003s) 2022-11-23T02:49:35.5368551Z 2022-11-23T02:49:35.5368814Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5368910Z Ran 1 test in 0.003s 2022-11-23T02:49:35.5368916Z 2022-11-23T02:49:35.5369009Z OK (skipped=1) 2022-11-23T02:49:35.5369015Z 2022-11-23T02:49:35.5369127Z Generating XML reports... 2022-11-23T02:49:35.5369556Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024512.xml 2022-11-23T02:49:35.5369867Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5370242Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5370414Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5370796Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5370969Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5370975Z 2022-11-23T02:49:35.5371071Z Running tests... 2022-11-23T02:49:35.5371335Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5371619Z test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86534 2022-11-23T02:49:35.5371822Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86535 2022-11-23T02:49:35.5372071Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5372443Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5372607Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5372985Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5373160Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5373379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5373747Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5373906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5374283Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5374458Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5374748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5375145Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5375528Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5375739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5376065Z STAGE:2022-11-23 02:45:19 86535:86535 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5376276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5376605Z STAGE:2022-11-23 02:45:19 86534:86534 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5376985Z STAGE:2022-11-23 02:45:19 86534:86534 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5377558Z STAGE:2022-11-23 02:45:19 86535:86535 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:45:19 86534:86534 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5377565Z 2022-11-23T02:49:35.5377915Z STAGE:2022-11-23 02:45:19 86535:86535 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5378241Z STAGE:2022-11-23 02:45:19 86535:86535 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5378564Z STAGE:2022-11-23 02:45:19 86534:86534 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5378895Z STAGE:2022-11-23 02:45:19 86534:86534 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5379223Z STAGE:2022-11-23 02:45:19 86535:86535 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5379572Z STAGE:2022-11-23 02:45:19 86534:86534 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5379922Z STAGE:2022-11-23 02:45:19 86535:86535 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5380011Z ok (5.221s) 2022-11-23T02:49:35.5380017Z 2022-11-23T02:49:35.5380282Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5380377Z Ran 1 test in 5.222s 2022-11-23T02:49:35.5380383Z 2022-11-23T02:49:35.5380464Z OK 2022-11-23T02:49:35.5380469Z 2022-11-23T02:49:35.5380580Z Generating XML reports... 2022-11-23T02:49:35.5381018Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024517.xml 2022-11-23T02:49:35.5381329Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5381700Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5381866Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5382248Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5382420Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5382426Z 2022-11-23T02:49:35.5382514Z Running tests... 2022-11-23T02:49:35.5382785Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5383023Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s) 2022-11-23T02:49:35.5383029Z 2022-11-23T02:49:35.5383292Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5383388Z Ran 1 test in 0.002s 2022-11-23T02:49:35.5383395Z 2022-11-23T02:49:35.5383488Z OK (skipped=1) 2022-11-23T02:49:35.5383493Z 2022-11-23T02:49:35.5383605Z Generating XML reports... 2022-11-23T02:49:35.5384049Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024526.xml 2022-11-23T02:49:35.5384421Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5384794Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5384955Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5385336Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5385510Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5385516Z 2022-11-23T02:49:35.5385612Z Running tests... 2022-11-23T02:49:35.5385874Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5386124Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s) 2022-11-23T02:49:35.5386178Z 2022-11-23T02:49:35.5386446Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5386543Z Ran 1 test in 0.002s 2022-11-23T02:49:35.5386549Z 2022-11-23T02:49:35.5386642Z OK (skipped=1) 2022-11-23T02:49:35.5386647Z 2022-11-23T02:49:35.5386759Z Generating XML reports... 2022-11-23T02:49:35.5387197Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024531.xml 2022-11-23T02:49:35.5387508Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5387878Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5388033Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5388414Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5388595Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5388601Z 2022-11-23T02:49:35.5388700Z Running tests... 2022-11-23T02:49:35.5388965Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5389262Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86879 2022-11-23T02:49:35.5389467Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86880 2022-11-23T02:49:35.5389721Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5390096Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5390257Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5390647Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5390824Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5391046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5391414Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5391574Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5391954Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5392127Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5392350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5392745Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5393198Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5393410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5393738Z STAGE:2022-11-23 02:45:38 86880:86880 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5393948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5394265Z STAGE:2022-11-23 02:45:38 86879:86879 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5394598Z STAGE:2022-11-23 02:45:38 86880:86880 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5394947Z STAGE:2022-11-23 02:45:38 86880:86880 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5395330Z STAGE:2022-11-23 02:45:38 86879:86879 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5395685Z STAGE:2022-11-23 02:45:38 86879:86879 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5396011Z STAGE:2022-11-23 02:45:38 86880:86880 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5396335Z STAGE:2022-11-23 02:45:38 86879:86879 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5396667Z STAGE:2022-11-23 02:45:38 86880:86880 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5397013Z STAGE:2022-11-23 02:45:38 86880:86880 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5397345Z STAGE:2022-11-23 02:45:38 86879:86879 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5397691Z STAGE:2022-11-23 02:45:38 86879:86879 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5397784Z ok (5.125s) 2022-11-23T02:49:35.5397793Z 2022-11-23T02:49:35.5398063Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5398160Z Ran 1 test in 5.126s 2022-11-23T02:49:35.5398166Z 2022-11-23T02:49:35.5398245Z OK 2022-11-23T02:49:35.5398251Z 2022-11-23T02:49:35.5398362Z Generating XML reports... 2022-11-23T02:49:35.5398800Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024535.xml 2022-11-23T02:49:35.5399112Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5399484Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5399646Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5400027Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5400205Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5400214Z 2022-11-23T02:49:35.5400310Z Running tests... 2022-11-23T02:49:35.5400565Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5400851Z test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87092 2022-11-23T02:49:35.5401053Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87093 2022-11-23T02:49:35.5401303Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5401672Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5401834Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5402216Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5402454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5402676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5403071Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5403444Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5403605Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5403984Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5404157Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5404376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5404839Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5405053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5405384Z STAGE:2022-11-23 02:45:47 87093:87093 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5405594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5405920Z STAGE:2022-11-23 02:45:47 87092:87092 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5406254Z STAGE:2022-11-23 02:45:47 87093:87093 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5406601Z STAGE:2022-11-23 02:45:47 87093:87093 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5406925Z STAGE:2022-11-23 02:45:47 87092:87092 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5407280Z STAGE:2022-11-23 02:45:47 87092:87092 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5407609Z STAGE:2022-11-23 02:45:47 87093:87093 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5408087Z STAGE:2022-11-23 02:45:47 87092:87092 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5408424Z STAGE:2022-11-23 02:45:47 87093:87093 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5408774Z STAGE:2022-11-23 02:45:47 87093:87093 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5409108Z STAGE:2022-11-23 02:45:47 87092:87092 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5409453Z STAGE:2022-11-23 02:45:47 87092:87092 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5409543Z ok (5.522s) 2022-11-23T02:49:35.5409549Z 2022-11-23T02:49:35.5409824Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5409922Z Ran 1 test in 5.522s 2022-11-23T02:49:35.5409927Z 2022-11-23T02:49:35.5410009Z OK 2022-11-23T02:49:35.5410015Z 2022-11-23T02:49:35.5410128Z Generating XML reports... 2022-11-23T02:49:35.5410565Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024544.xml 2022-11-23T02:49:35.5410878Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5411249Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5411408Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5411786Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5411967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5412047Z 2022-11-23T02:49:35.5412144Z Running tests... 2022-11-23T02:49:35.5412412Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5412707Z test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87305 2022-11-23T02:49:35.5412913Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87306 2022-11-23T02:49:35.5413154Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5413524Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5413687Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5414064Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5414296Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5414522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5414895Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5415070Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5415449Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5415623Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5415842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5416235Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5416630Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5416840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5417097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5417186Z ok (5.424s) 2022-11-23T02:49:35.5417192Z 2022-11-23T02:49:35.5417457Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5417554Z Ran 1 test in 5.425s 2022-11-23T02:49:35.5417559Z 2022-11-23T02:49:35.5417641Z OK 2022-11-23T02:49:35.5417646Z 2022-11-23T02:49:35.5417757Z Generating XML reports... 2022-11-23T02:49:35.5418198Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024554.xml 2022-11-23T02:49:35.5418512Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5418881Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5419057Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5419438Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5419613Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5419618Z 2022-11-23T02:49:35.5419717Z Running tests... 2022-11-23T02:49:35.5419980Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5420276Z test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87512 2022-11-23T02:49:35.5420477Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87513 2022-11-23T02:49:35.5420729Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5421175Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5421336Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5421722Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5421894Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5422115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5422485Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5422646Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5423073Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5423255Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5423479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5423876Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5424268Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5424478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5424805Z STAGE:2022-11-23 02:46:06 87513:87513 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5425005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5425353Z STAGE:2022-11-23 02:46:07 87512:87512 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5425689Z STAGE:2022-11-23 02:46:07 87512:87512 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5426034Z STAGE:2022-11-23 02:46:07 87513:87513 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5426380Z STAGE:2022-11-23 02:46:07 87512:87512 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5426725Z STAGE:2022-11-23 02:46:07 87513:87513 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5427050Z STAGE:2022-11-23 02:46:07 87513:87513 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5427377Z STAGE:2022-11-23 02:46:07 87512:87512 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5427709Z STAGE:2022-11-23 02:46:07 87513:87513 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5428044Z STAGE:2022-11-23 02:46:07 87512:87512 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5428392Z STAGE:2022-11-23 02:46:07 87513:87513 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5428739Z STAGE:2022-11-23 02:46:07 87512:87512 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5428828Z ok (4.914s) 2022-11-23T02:49:35.5428834Z 2022-11-23T02:49:35.5429159Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5429258Z Ran 1 test in 4.914s 2022-11-23T02:49:35.5429263Z 2022-11-23T02:49:35.5429346Z OK 2022-11-23T02:49:35.5429352Z 2022-11-23T02:49:35.5429462Z Generating XML reports... 2022-11-23T02:49:35.5429901Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024604.xml 2022-11-23T02:49:35.5430213Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5430653Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5430819Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5431213Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5431387Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5431393Z 2022-11-23T02:49:35.5431489Z Running tests... 2022-11-23T02:49:35.5431745Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5431981Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s) 2022-11-23T02:49:35.5431989Z 2022-11-23T02:49:35.5432252Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5432346Z Ran 1 test in 0.002s 2022-11-23T02:49:35.5432352Z 2022-11-23T02:49:35.5432444Z OK (skipped=1) 2022-11-23T02:49:35.5432454Z 2022-11-23T02:49:35.5432644Z Generating XML reports... 2022-11-23T02:49:35.5433088Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024613.xml 2022-11-23T02:49:35.5433411Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5433781Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5433940Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5434320Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5434495Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5434501Z 2022-11-23T02:49:35.5434600Z Running tests... 2022-11-23T02:49:35.5434864Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5435120Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s) 2022-11-23T02:49:35.5435127Z 2022-11-23T02:49:35.5435389Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5435486Z Ran 1 test in 0.002s 2022-11-23T02:49:35.5435491Z 2022-11-23T02:49:35.5435583Z OK (skipped=1) 2022-11-23T02:49:35.5435589Z 2022-11-23T02:49:35.5435700Z Generating XML reports... 2022-11-23T02:49:35.5436152Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024618.xml 2022-11-23T02:49:35.5436466Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5436835Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5436987Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5437375Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5437550Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5437555Z 2022-11-23T02:49:35.5437651Z Running tests... 2022-11-23T02:49:35.5437917Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5438214Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87857 2022-11-23T02:49:35.5438432Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87858 2022-11-23T02:49:35.5438682Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5439051Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5439215Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5439659Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5439835Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5440058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5440428Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5440590Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5440971Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5441149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5441371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5441815Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5442225Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5442438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5442650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5442870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:49:35.5443080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:49:35.5443471Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.5443864Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:49:35.5444195Z STAGE:2022-11-23 02:46:24 87857:87857 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5444533Z STAGE:2022-11-23 02:46:24 87858:87858 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5444866Z STAGE:2022-11-23 02:46:24 87857:87857 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5445214Z STAGE:2022-11-23 02:46:24 87857:87857 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5445540Z STAGE:2022-11-23 02:46:24 87857:87857 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5445872Z STAGE:2022-11-23 02:46:24 87858:87858 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5446221Z STAGE:2022-11-23 02:46:24 87858:87858 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5446555Z STAGE:2022-11-23 02:46:24 87858:87858 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5446887Z STAGE:2022-11-23 02:46:24 87858:87858 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5447246Z STAGE:2022-11-23 02:46:24 87858:87858 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5447578Z STAGE:2022-11-23 02:46:24 87857:87857 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5447968Z STAGE:2022-11-23 02:46:24 87857:87857 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5448057Z ok (5.024s) 2022-11-23T02:49:35.5448063Z 2022-11-23T02:49:35.5448328Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5448426Z Ran 1 test in 5.024s 2022-11-23T02:49:35.5448432Z 2022-11-23T02:49:35.5448511Z OK 2022-11-23T02:49:35.5448517Z 2022-11-23T02:49:35.5448628Z Generating XML reports... 2022-11-23T02:49:35.5449145Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024622.xml 2022-11-23T02:49:35.5449454Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5449825Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5449988Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5450362Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5450551Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5450568Z 2022-11-23T02:49:35.5450656Z Running tests... 2022-11-23T02:49:35.5450924Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5451269Z test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88076 2022-11-23T02:49:35.5451484Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88077 2022-11-23T02:49:35.5451735Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5452111Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5452286Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5452667Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5452841Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5453059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5453433Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5453596Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5453976Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5454152Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5454370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5454764Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5455167Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5455381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5455594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5455743Z skip: Skipped due to small world size. (5.328s) 2022-11-23T02:49:35.5455749Z 2022-11-23T02:49:35.5456013Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5456101Z Ran 1 test in 5.329s 2022-11-23T02:49:35.5456116Z 2022-11-23T02:49:35.5456201Z OK (skipped=1) 2022-11-23T02:49:35.5456207Z 2022-11-23T02:49:35.5456318Z Generating XML reports... 2022-11-23T02:49:35.5456755Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024631.xml 2022-11-23T02:49:35.5457068Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5457441Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5457601Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5458043Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5458252Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5458258Z 2022-11-23T02:49:35.5458354Z Running tests... 2022-11-23T02:49:35.5458619Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5458922Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88283 2022-11-23T02:49:35.5459128Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88284 2022-11-23T02:49:35.5459378Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5459762Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5459976Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5460362Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5460535Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5460755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5461126Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5461286Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5461670Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5461856Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5462069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5462470Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5462859Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5463071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5463283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5463384Z ok (5.014s) 2022-11-23T02:49:35.5463390Z 2022-11-23T02:49:35.5463652Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5463749Z Ran 1 test in 5.015s 2022-11-23T02:49:35.5463755Z 2022-11-23T02:49:35.5463835Z OK 2022-11-23T02:49:35.5463840Z 2022-11-23T02:49:35.5463951Z Generating XML reports... 2022-11-23T02:49:35.5464389Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024641.xml 2022-11-23T02:49:35.5464715Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5465084Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5465244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5465626Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5465801Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5465807Z 2022-11-23T02:49:35.5465906Z Running tests... 2022-11-23T02:49:35.5466183Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5466469Z test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88490 2022-11-23T02:49:35.5466735Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88491 2022-11-23T02:49:35.5466991Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5467365Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5467529Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5467898Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5468085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5468309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5468678Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5468893Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5469277Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5469452Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5469670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5470061Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5470466Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5470677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5470887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5470983Z ok (4.921s) 2022-11-23T02:49:35.5470989Z 2022-11-23T02:49:35.5471252Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5471349Z Ran 1 test in 4.921s 2022-11-23T02:49:35.5471355Z 2022-11-23T02:49:35.5471435Z OK 2022-11-23T02:49:35.5471441Z 2022-11-23T02:49:35.5471552Z Generating XML reports... 2022-11-23T02:49:35.5472007Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024650.xml 2022-11-23T02:49:35.5472320Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5472691Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5472852Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5473232Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5473403Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5473417Z 2022-11-23T02:49:35.5473505Z Running tests... 2022-11-23T02:49:35.5473781Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5474083Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88697 2022-11-23T02:49:35.5474286Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88698 2022-11-23T02:49:35.5474536Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5474906Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5475067Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5475449Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5475708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5475928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5476329Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5476697Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5476860Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5477242Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5477414Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5477691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5478095Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5478306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5478515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5478603Z ok (5.028s) 2022-11-23T02:49:35.5478609Z 2022-11-23T02:49:35.5478872Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5478968Z Ran 1 test in 5.028s 2022-11-23T02:49:35.5478974Z 2022-11-23T02:49:35.5479046Z OK 2022-11-23T02:49:35.5479062Z 2022-11-23T02:49:35.5479165Z Generating XML reports... 2022-11-23T02:49:35.5479602Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024659.xml 2022-11-23T02:49:35.5479928Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5480303Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5480464Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5480845Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5481019Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5481025Z 2022-11-23T02:49:35.5481125Z Running tests... 2022-11-23T02:49:35.5481419Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5481746Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88904 2022-11-23T02:49:35.5481948Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88905 2022-11-23T02:49:35.5482209Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5482578Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5482739Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5483135Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5483312Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5483532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5483900Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5484061Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5484507Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5484681Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5484902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5485285Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5485686Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5485897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5486108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5486436Z STAGE:2022-11-23 02:47:11 88904:88904 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5486811Z STAGE:2022-11-23 02:47:11 88905:88905 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5487161Z STAGE:2022-11-23 02:47:11 88904:88904 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5487762Z STAGE:2022-11-23 02:47:11 88905:88905 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:47:11 88904:88904 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5487770Z 2022-11-23T02:49:35.5488121Z STAGE:2022-11-23 02:47:11 88905:88905 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5488211Z ok (5.573s) 2022-11-23T02:49:35.5488217Z 2022-11-23T02:49:35.5488482Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5488580Z Ran 1 test in 5.574s 2022-11-23T02:49:35.5488586Z 2022-11-23T02:49:35.5488666Z OK 2022-11-23T02:49:35.5488672Z 2022-11-23T02:49:35.5488785Z Generating XML reports... 2022-11-23T02:49:35.5489234Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024708.xml 2022-11-23T02:49:35.5489545Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5489928Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5490090Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5490470Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5490642Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5490649Z 2022-11-23T02:49:35.5490746Z Running tests... 2022-11-23T02:49:35.5491011Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5491334Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89117 2022-11-23T02:49:35.5491576Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89118 2022-11-23T02:49:35.5491827Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5492197Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5492348Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5492726Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5492903Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5493138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5493584Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5493746Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5494125Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5494303Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5494525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5494933Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5495324Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5495537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5495810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5496145Z STAGE:2022-11-23 02:47:20 89117:89117 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5496486Z STAGE:2022-11-23 02:47:20 89118:89118 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5496819Z STAGE:2022-11-23 02:47:20 89117:89117 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5497172Z STAGE:2022-11-23 02:47:20 89117:89117 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5497506Z STAGE:2022-11-23 02:47:20 89118:89118 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5497865Z STAGE:2022-11-23 02:47:20 89118:89118 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5497955Z ok (4.865s) 2022-11-23T02:49:35.5497961Z 2022-11-23T02:49:35.5498233Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5498332Z Ran 1 test in 4.866s 2022-11-23T02:49:35.5498338Z 2022-11-23T02:49:35.5498410Z OK 2022-11-23T02:49:35.5498426Z 2022-11-23T02:49:35.5498529Z Generating XML reports... 2022-11-23T02:49:35.5498971Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024718.xml 2022-11-23T02:49:35.5499281Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5499664Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5499826Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5500212Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5500385Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5500394Z 2022-11-23T02:49:35.5500493Z Running tests... 2022-11-23T02:49:35.5500758Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5501070Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89330 2022-11-23T02:49:35.5501275Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89331 2022-11-23T02:49:35.5501526Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5501897Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5502057Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5502450Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5502683Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5502905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5503300Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5503675Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5503835Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5504213Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5504387Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5504600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5505045Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5505264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5505606Z STAGE:2022-11-23 02:47:30 89331:89331 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5505818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5506147Z STAGE:2022-11-23 02:47:30 89330:89330 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5506481Z STAGE:2022-11-23 02:47:30 89331:89331 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5506842Z STAGE:2022-11-23 02:47:30 89331:89331 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5507175Z STAGE:2022-11-23 02:47:30 89330:89330 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5507528Z STAGE:2022-11-23 02:47:30 89330:89330 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5507618Z ok (5.047s) 2022-11-23T02:49:35.5507624Z 2022-11-23T02:49:35.5507887Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5507985Z Ran 1 test in 5.047s 2022-11-23T02:49:35.5507991Z 2022-11-23T02:49:35.5508072Z OK 2022-11-23T02:49:35.5508078Z 2022-11-23T02:49:35.5508199Z Generating XML reports... 2022-11-23T02:49:35.5508640Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024727.xml 2022-11-23T02:49:35.5508952Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5509323Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5509484Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5509872Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5510059Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5510066Z 2022-11-23T02:49:35.5510162Z Running tests... 2022-11-23T02:49:35.5510428Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5510638Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.001s) 2022-11-23T02:49:35.5510656Z 2022-11-23T02:49:35.5510910Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5511006Z Ran 1 test in 0.002s 2022-11-23T02:49:35.5511012Z 2022-11-23T02:49:35.5511109Z OK (skipped=1) 2022-11-23T02:49:35.5511115Z 2022-11-23T02:49:35.5511229Z Generating XML reports... 2022-11-23T02:49:35.5511668Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024736.xml 2022-11-23T02:49:35.5512063Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5512441Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5512602Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5512984Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5513157Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5513163Z 2022-11-23T02:49:35.5513259Z Running tests... 2022-11-23T02:49:35.5513522Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5513768Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s) 2022-11-23T02:49:35.5513774Z 2022-11-23T02:49:35.5514034Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5514183Z Ran 1 test in 0.002s 2022-11-23T02:49:35.5514190Z 2022-11-23T02:49:35.5514284Z OK (skipped=1) 2022-11-23T02:49:35.5514290Z 2022-11-23T02:49:35.5514402Z Generating XML reports... 2022-11-23T02:49:35.5514851Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024740.xml 2022-11-23T02:49:35.5515207Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5515583Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5515747Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5516128Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5516295Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5516315Z 2022-11-23T02:49:35.5516406Z Running tests... 2022-11-23T02:49:35.5516671Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5516928Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s) 2022-11-23T02:49:35.5516934Z 2022-11-23T02:49:35.5517194Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5517290Z Ran 1 test in 0.002s 2022-11-23T02:49:35.5517296Z 2022-11-23T02:49:35.5517392Z OK (skipped=1) 2022-11-23T02:49:35.5517397Z 2022-11-23T02:49:35.5517509Z Generating XML reports... 2022-11-23T02:49:35.5517944Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024744.xml 2022-11-23T02:49:35.5518255Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5518641Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5518807Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5519190Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5519367Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5519373Z 2022-11-23T02:49:35.5519469Z Running tests... 2022-11-23T02:49:35.5519735Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5520043Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89741 2022-11-23T02:49:35.5520247Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89742 2022-11-23T02:49:35.5520499Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5520934Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5521094Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5521487Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5521661Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5521874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5522239Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5522401Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5522781Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5523003Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5523244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5523640Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5524029Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5524240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5524568Z STAGE:2022-11-23 02:47:51 89742:89742 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5524789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5525116Z STAGE:2022-11-23 02:47:51 89741:89741 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5525458Z STAGE:2022-11-23 02:47:51 89742:89742 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5525810Z STAGE:2022-11-23 02:47:51 89742:89742 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5526154Z STAGE:2022-11-23 02:47:51 89741:89741 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5526500Z STAGE:2022-11-23 02:47:51 89741:89741 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5526589Z ok (5.087s) 2022-11-23T02:49:35.5526595Z 2022-11-23T02:49:35.5526857Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5526956Z Ran 1 test in 5.087s 2022-11-23T02:49:35.5526962Z 2022-11-23T02:49:35.5527045Z OK 2022-11-23T02:49:35.5527050Z 2022-11-23T02:49:35.5527161Z Generating XML reports... 2022-11-23T02:49:35.5527611Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024748.xml 2022-11-23T02:49:35.5528059Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5528425Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5528586Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5528970Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5529148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5529154Z 2022-11-23T02:49:35.5529263Z Running tests... 2022-11-23T02:49:35.5529529Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5529825Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89954 2022-11-23T02:49:35.5530034Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89955 2022-11-23T02:49:35.5530357Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5530731Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5530893Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5531272Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5531446Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5531666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5532047Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5532206Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5532642Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5532821Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5533046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5533444Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5533846Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5534065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5534276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5534357Z ok (4.968s) 2022-11-23T02:49:35.5534363Z 2022-11-23T02:49:35.5534636Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5534733Z Ran 1 test in 4.968s 2022-11-23T02:49:35.5534739Z 2022-11-23T02:49:35.5534819Z OK 2022-11-23T02:49:35.5534825Z 2022-11-23T02:49:35.5534934Z Generating XML reports... 2022-11-23T02:49:35.5535373Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024758.xml 2022-11-23T02:49:35.5535683Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5536054Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5536214Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5536608Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5536787Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5536800Z 2022-11-23T02:49:35.5536897Z Running tests... 2022-11-23T02:49:35.5537163Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5537486Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90161 2022-11-23T02:49:35.5537689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90162 2022-11-23T02:49:35.5537942Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5538326Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5538487Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5538869Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5539107Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5539327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5539711Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5539863Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5540243Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5540419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5540641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5541030Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5541499Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5541712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5542048Z STAGE:2022-11-23 02:48:10 90162:90162 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5542260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5542587Z STAGE:2022-11-23 02:48:10 90161:90161 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5542930Z STAGE:2022-11-23 02:48:10 90161:90161 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5543280Z STAGE:2022-11-23 02:48:10 90161:90161 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5543611Z STAGE:2022-11-23 02:48:10 90162:90162 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5543967Z STAGE:2022-11-23 02:48:10 90162:90162 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5544058Z ok (5.474s) 2022-11-23T02:49:35.5544064Z 2022-11-23T02:49:35.5544342Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5544437Z Ran 1 test in 5.475s 2022-11-23T02:49:35.5544443Z 2022-11-23T02:49:35.5544522Z OK 2022-11-23T02:49:35.5544527Z 2022-11-23T02:49:35.5544636Z Generating XML reports... 2022-11-23T02:49:35.5545074Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024807.xml 2022-11-23T02:49:35.5545386Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5545767Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5545928Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5546307Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5546482Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5546488Z 2022-11-23T02:49:35.5546584Z Running tests... 2022-11-23T02:49:35.5546848Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5547165Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90374 2022-11-23T02:49:35.5547371Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90375 2022-11-23T02:49:35.5547621Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5547991Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5548218Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5548611Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5548787Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5549009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5549379Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5549541Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5549923Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5550125Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5550411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5550815Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5551212Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5551424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5551634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5551964Z STAGE:2022-11-23 02:48:19 90374:90374 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5552291Z STAGE:2022-11-23 02:48:19 90375:90375 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:49:35.5552627Z STAGE:2022-11-23 02:48:19 90374:90374 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5552972Z STAGE:2022-11-23 02:48:19 90374:90374 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5553307Z STAGE:2022-11-23 02:48:19 90375:90375 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:49:35.5553673Z STAGE:2022-11-23 02:48:19 90375:90375 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:49:35.5553762Z ok (4.989s) 2022-11-23T02:49:35.5553768Z 2022-11-23T02:49:35.5554034Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5554136Z Ran 1 test in 4.990s 2022-11-23T02:49:35.5554142Z 2022-11-23T02:49:35.5554222Z OK 2022-11-23T02:49:35.5554227Z 2022-11-23T02:49:35.5554339Z Generating XML reports... 2022-11-23T02:49:35.5554778Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024816.xml 2022-11-23T02:49:35.5555118Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5555499Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5555662Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5556044Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5556220Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5556226Z 2022-11-23T02:49:35.5556322Z Running tests... 2022-11-23T02:49:35.5556587Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5556922Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90587 2022-11-23T02:49:35.5557125Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90588 2022-11-23T02:49:35.5557385Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5557813Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5557975Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5558380Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5558545Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5558767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5559137Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5559300Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5559740Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5559925Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5560177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5560575Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5560967Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5561179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5561387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5561479Z ok (5.176s) 2022-11-23T02:49:35.5561485Z 2022-11-23T02:49:35.5561773Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5561874Z Ran 1 test in 5.176s 2022-11-23T02:49:35.5561885Z 2022-11-23T02:49:35.5561968Z OK 2022-11-23T02:49:35.5561974Z 2022-11-23T02:49:35.5562085Z Generating XML reports... 2022-11-23T02:49:35.5562520Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024826.xml 2022-11-23T02:49:35.5562832Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5563229Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5563392Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5563774Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5563951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5563956Z 2022-11-23T02:49:35.5564052Z Running tests... 2022-11-23T02:49:35.5564315Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5564625Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90980 2022-11-23T02:49:35.5564829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90981 2022-11-23T02:49:35.5565090Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5565462Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5565624Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5566005Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5566179Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5566478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5566851Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5567014Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5567395Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5567569Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5567836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5568235Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5568626Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5568902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5569131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5569220Z ok (5.478s) 2022-11-23T02:49:35.5569226Z 2022-11-23T02:49:35.5569495Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5569591Z Ran 1 test in 5.479s 2022-11-23T02:49:35.5569597Z 2022-11-23T02:49:35.5569681Z OK 2022-11-23T02:49:35.5569686Z 2022-11-23T02:49:35.5569788Z Generating XML reports... 2022-11-23T02:49:35.5570229Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024835.xml 2022-11-23T02:49:35.5570543Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5570938Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5571108Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5571489Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5571665Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5571671Z 2022-11-23T02:49:35.5571768Z Running tests... 2022-11-23T02:49:35.5572031Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5572352Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91375 2022-11-23T02:49:35.5572560Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91376 2022-11-23T02:49:35.5572811Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5573184Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5573348Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5573736Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5573909Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5574127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5574500Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5574660Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5575041Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5575241Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5575527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5575923Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5576304Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5576517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5576732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5576821Z ok (7.180s) 2022-11-23T02:49:35.5576827Z 2022-11-23T02:49:35.5577113Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5577212Z Ran 1 test in 7.180s 2022-11-23T02:49:35.5577217Z 2022-11-23T02:49:35.5577299Z OK 2022-11-23T02:49:35.5577308Z 2022-11-23T02:49:35.5577468Z Generating XML reports... 2022-11-23T02:49:35.5577917Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024845.xml 2022-11-23T02:49:35.5578229Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5578607Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5578769Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5579167Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5579345Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5579351Z 2022-11-23T02:49:35.5579448Z Running tests... 2022-11-23T02:49:35.5579715Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5580043Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91592 2022-11-23T02:49:35.5580248Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91593 2022-11-23T02:49:35.5580497Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5580869Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5581030Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5581429Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5581604Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5581819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5582191Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5582369Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5582773Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5582950Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5583175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5583567Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5583962Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5584173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5584470Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjn0zhr5h 2022-11-23T02:49:35.5584734Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjn0zhr5h/_remote_module_non_scriptable.py 2022-11-23T02:49:35.5584946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5585180Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqfcxmo3o 2022-11-23T02:49:35.5585428Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqfcxmo3o/_remote_module_non_scriptable.py 2022-11-23T02:49:35.5585519Z ok (5.074s) 2022-11-23T02:49:35.5585525Z 2022-11-23T02:49:35.5585803Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5585900Z Ran 1 test in 5.075s 2022-11-23T02:49:35.5585905Z 2022-11-23T02:49:35.5585986Z OK 2022-11-23T02:49:35.5585992Z 2022-11-23T02:49:35.5586105Z Generating XML reports... 2022-11-23T02:49:35.5586602Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024856.xml 2022-11-23T02:49:35.5586918Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5587297Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5587450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5587833Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5588030Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5588036Z 2022-11-23T02:49:35.5588132Z Running tests... 2022-11-23T02:49:35.5588397Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5588695Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91865 2022-11-23T02:49:35.5588911Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91866 2022-11-23T02:49:35.5589163Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5589534Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5589696Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5590080Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5590256Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5590486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5590859Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5591024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5591410Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5591585Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5591806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5592210Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5592609Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5592821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5593042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5593334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmdzigahj 2022-11-23T02:49:35.5593582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmdzigahj/_remote_module_non_scriptable.py 2022-11-23T02:49:35.5593804Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplg4s3u58 2022-11-23T02:49:35.5594053Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplg4s3u58/_remote_module_non_scriptable.py 2022-11-23T02:49:35.5594156Z ok (5.363s) 2022-11-23T02:49:35.5594161Z 2022-11-23T02:49:35.5594431Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5594530Z Ran 1 test in 5.364s 2022-11-23T02:49:35.5594536Z 2022-11-23T02:49:35.5594616Z OK 2022-11-23T02:49:35.5594622Z 2022-11-23T02:49:35.5594736Z Generating XML reports... 2022-11-23T02:49:35.5595232Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024905.xml 2022-11-23T02:49:35.5595555Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5595927Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5596090Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5596476Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5596650Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5596656Z 2022-11-23T02:49:35.5596767Z Running tests... 2022-11-23T02:49:35.5597041Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5597377Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92072 2022-11-23T02:49:35.5597584Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92073 2022-11-23T02:49:35.5597837Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:49:35.5598211Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5598380Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5598760Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5598934Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5599147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:49:35.5599521Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5599693Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5600074Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5600264Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5600495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:49:35.5600887Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5601278Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:49:35.5601491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:49:35.5601716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:49:35.5602013Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi7u38zf7 2022-11-23T02:49:35.5602259Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi7u38zf7/_remote_module_non_scriptable.py 2022-11-23T02:49:35.5602523Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv1lqooih 2022-11-23T02:49:35.5602773Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv1lqooih/_remote_module_non_scriptable.py 2022-11-23T02:49:35.5603578Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:49:35.5604352Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:49:35.5604447Z ok (7.477s) 2022-11-23T02:49:35.5604454Z 2022-11-23T02:49:35.5604727Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5604829Z Ran 1 test in 7.478s 2022-11-23T02:49:35.5604835Z 2022-11-23T02:49:35.5604915Z OK 2022-11-23T02:49:35.5604921Z 2022-11-23T02:49:35.5605039Z Generating XML reports... 2022-11-23T02:49:35.5605485Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024915.xml 2022-11-23T02:49:35.5605799Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5606178Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5606342Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5606733Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5606931Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5606936Z 2022-11-23T02:49:35.5607036Z Running tests... 2022-11-23T02:49:35.5607309Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5607819Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.001s) 2022-11-23T02:49:35.5607829Z 2022-11-23T02:49:35.5608093Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5608198Z Ran 1 test in 0.002s 2022-11-23T02:49:35.5608203Z 2022-11-23T02:49:35.5608301Z OK (skipped=1) 2022-11-23T02:49:35.5608307Z 2022-11-23T02:49:35.5608409Z Generating XML reports... 2022-11-23T02:49:35.5608853Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024926.xml 2022-11-23T02:49:35.5609170Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-11-23T02:49:35.5609554Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:49:35.5609714Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:49:35.5610101Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:49:35.5610384Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:49:35.5610392Z 2022-11-23T02:49:35.5610500Z Running tests... 2022-11-23T02:49:35.5610774Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5611252Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.002s) 2022-11-23T02:49:35.5611258Z 2022-11-23T02:49:35.5611522Z ---------------------------------------------------------------------- 2022-11-23T02:49:35.5611636Z Ran 1 test in 0.002s 2022-11-23T02:49:35.5611641Z 2022-11-23T02:49:35.5611738Z OK (skipped=1) 2022-11-23T02:49:35.5611743Z 2022-11-23T02:49:35.5611870Z Generating XML reports... 2022-11-23T02:49:35.5612308Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123024931.xml 2022-11-23T02:49:35.5612318Z 2022-11-23T02:49:35.5613000Z ##[endgroup] 2022-11-23T02:49:35.5613475Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_distributed_spawn_xw60i0rv) 2022-11-23T02:49:35.5613482Z 2022-11-23T02:49:35.5613744Z Running distributed/test_pg_wrapper ... [2022-11-23 02:49:35.295223] 2022-11-23T02:49:35.5614248Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_pg_wrapper.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:49:35.295619] 2022-11-23T02:51:45.2749694Z 2022-11-23T02:51:45.2751156Z Expand the folded group to see the log file of distributed/test_pg_wrapper 2022-11-23T02:51:45.2753509Z ##[group]PRINTING LOG FILE of distributed/test_pg_wrapper (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_pg_wrapper_7px1l7zb) 2022-11-23T02:51:45.2754874Z 2022-11-23T02:51:45.2755804Z 2022-11-23T02:51:45.2761059Z , <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_cuda>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_cuda_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_cuda>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_cuda_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_debug_mode>]> 2022-11-23T02:51:45.2765752Z test_collective_hang (__main__.ProcessGroupGlooWrapperTest) 2022-11-23T02:51:45.2767114Z test_collective_shape_mismatch (__main__.ProcessGroupGlooWrapperTest) 2022-11-23T02:51:45.2769210Z test_collective_shape_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest) 2022-11-23T02:51:45.2770771Z test_collective_shape_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest) 2022-11-23T02:51:45.2772398Z test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest) 2022-11-23T02:51:45.2774087Z test_collectives_op_mismatch (__main__.ProcessGroupGlooWrapperTest) 2022-11-23T02:51:45.2775667Z test_collectives_op_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest) 2022-11-23T02:51:45.2777101Z test_collectives_op_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest) 2022-11-23T02:51:45.2778601Z test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest) 2022-11-23T02:51:45.2781637Z , <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_shape_mismatch>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_shape_mismatch_debug_mode>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collectives_op_mismatch>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collectives_op_mismatch_debug_mode>]> 2022-11-23T02:51:45.2785760Z test_collective_hang (__main__.ProcessGroupNCCLWrapperTest) 2022-11-23T02:51:45.2787133Z test_collective_shape_mismatch (__main__.ProcessGroupNCCLWrapperTest) 2022-11-23T02:51:45.2788713Z test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest) 2022-11-23T02:51:45.2790192Z test_collectives_op_mismatch (__main__.ProcessGroupNCCLWrapperTest) 2022-11-23T02:51:45.2791668Z test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest) 2022-11-23T02:51:45.2793974Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-11-23T02:51:45.2796214Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.2797995Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.2800134Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.2801648Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.2804151Z 2022-11-23T02:51:45.2804459Z Running tests... 2022-11-23T02:51:45.2805721Z ---------------------------------------------------------------------- 2022-11-23T02:51:45.2807284Z test_collective_hang (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92487 2022-11-23T02:51:45.2808938Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92488 2022-11-23T02:51:45.2810114Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 92489 2022-11-23T02:51:45.2811263Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 92490 2022-11-23T02:51:45.2812941Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.2814125Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.2815687Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.2816889Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.2818020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:45.2819669Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.2820801Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.2822350Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.2823556Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.2824702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:45.2826370Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.2827529Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.2829096Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.2830282Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.2831413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:45.2833065Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.2834208Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.2835759Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.2837234Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.2838359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:45.2839615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:45.2840859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:45.2842119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T02:51:45.2843888Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:45.2845239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T02:51:45.2847157Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:45.2849461Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:45.2851308Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:45.2852646Z [E ProcessGroupGloo.cpp:2802] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms 2022-11-23T02:51:45.2853835Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 2000 ms 2022-11-23T02:51:45.2855352Z [E ProcessGroupGloo.cpp:137] Rank 2 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank. 2022-11-23T02:51:45.2857162Z [E ProcessGroupGloo.cpp:137] Rank 3 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank. 2022-11-23T02:51:45.2858335Z ok (4.920s) 2022-11-23T02:51:45.2858665Z 2022-11-23T02:51:45.2859236Z ---------------------------------------------------------------------- 2022-11-23T02:51:45.2859893Z Ran 1 test in 4.920s 2022-11-23T02:51:45.2860204Z 2022-11-23T02:51:45.2860377Z OK 2022-11-23T02:51:45.2860641Z 2022-11-23T02:51:45.2860861Z Generating XML reports... 2022-11-23T02:51:45.2862207Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123024938.xml 2022-11-23T02:51:45.2863601Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-11-23T02:51:45.2864917Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.2865822Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.2867059Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.2868027Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.2868483Z 2022-11-23T02:51:45.2868686Z Running tests... 2022-11-23T02:51:45.2869497Z ---------------------------------------------------------------------- 2022-11-23T02:51:45.2870586Z test_collective_shape_mismatch (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92858 2022-11-23T02:51:45.2871681Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92859 2022-11-23T02:51:45.2872574Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 92860 2022-11-23T02:51:45.2873458Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 92861 2022-11-23T02:51:45.2874700Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.2875725Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.2876889Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.2877816Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.2878664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:45.2879921Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.2880790Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.2881960Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.2882873Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.2883708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:45.2885073Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.2885952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.2887128Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.2888349Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.2889241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:45.2890576Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.2891499Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.2892726Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.2893716Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.2894625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:45.2895626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:45.2896647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:45.2897656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T02:51:45.2898525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T02:51:45.2899218Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:45.2899900Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:45.2900589Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:45.2901268Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:45.2901643Z ok (5.414s) 2022-11-23T02:51:45.2901782Z 2022-11-23T02:51:45.2902056Z ---------------------------------------------------------------------- 2022-11-23T02:51:45.2902377Z Ran 1 test in 5.415s 2022-11-23T02:51:45.2902530Z 2022-11-23T02:51:45.2902616Z OK 2022-11-23T02:51:45.2902729Z 2022-11-23T02:51:45.2902846Z Generating XML reports... 2022-11-23T02:51:45.2903468Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123024947.xml 2022-11-23T02:51:45.2904128Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-11-23T02:51:45.2904750Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.2905266Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.2905847Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.2906305Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.2906524Z 2022-11-23T02:51:45.2906613Z Running tests... 2022-11-23T02:51:45.2907025Z ---------------------------------------------------------------------- 2022-11-23T02:51:45.2907569Z test_collective_shape_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93229 2022-11-23T02:51:45.2908121Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93230 2022-11-23T02:51:45.2908562Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 93231 2022-11-23T02:51:45.2909061Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 93232 2022-11-23T02:51:45.2909685Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.2910109Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.2910688Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.2911148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.2911575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:45.2912192Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.2912631Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.2913212Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.2913660Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.2914089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:45.2914706Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.2915144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.2915720Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.2916180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.2916606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:45.2917225Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.2917655Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.2918233Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.2918691Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.2919117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:45.2919503Z skip: Need at least 4 CUDA devices (4.726s) 2022-11-23T02:51:45.2919682Z 2022-11-23T02:51:45.2919952Z ---------------------------------------------------------------------- 2022-11-23T02:51:45.2920272Z Ran 1 test in 4.727s 2022-11-23T02:51:45.2920411Z 2022-11-23T02:51:45.2920510Z OK (skipped=1) 2022-11-23T02:51:45.2920654Z 2022-11-23T02:51:45.2920772Z Generating XML reports... 2022-11-23T02:51:45.2921391Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123024956.xml 2022-11-23T02:51:45.2922128Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-11-23T02:51:45.2922750Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.2923190Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.2923766Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.2924209Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.2924426Z 2022-11-23T02:51:45.2924525Z Running tests... 2022-11-23T02:51:45.2924936Z ---------------------------------------------------------------------- 2022-11-23T02:51:45.2925489Z test_collective_shape_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93564 2022-11-23T02:51:45.2926137Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93565 2022-11-23T02:51:45.2926662Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 93566 2022-11-23T02:51:45.2927189Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 93567 2022-11-23T02:51:45.2928005Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.2928535Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.2929231Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.2929778Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.2930291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:45.2931042Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.2931575Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.2932257Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.2932803Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.2933317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:45.2934056Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.2934578Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.2935278Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.2935828Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.2936342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:45.2937078Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.2937603Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.2938299Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.2938818Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.2939242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:45.2939629Z skip: Need at least 4 CUDA devices (5.091s) 2022-11-23T02:51:45.2939807Z 2022-11-23T02:51:45.2940081Z ---------------------------------------------------------------------- 2022-11-23T02:51:45.2940389Z Ran 1 test in 5.092s 2022-11-23T02:51:45.2940541Z 2022-11-23T02:51:45.2940639Z OK (skipped=1) 2022-11-23T02:51:45.2940857Z 2022-11-23T02:51:45.2940979Z Generating XML reports... 2022-11-23T02:51:45.2941599Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123025005.xml 2022-11-23T02:51:45.2942253Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-11-23T02:51:45.2942873Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.2943314Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.2943879Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.2944337Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.2944555Z 2022-11-23T02:51:45.2944657Z Running tests... 2022-11-23T02:51:45.2945068Z ---------------------------------------------------------------------- 2022-11-23T02:51:45.2945790Z test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93899 2022-11-23T02:51:45.2946358Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93900 2022-11-23T02:51:45.2946799Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 93901 2022-11-23T02:51:45.2947224Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 93902 2022-11-23T02:51:45.2947841Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.2948283Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.2948868Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.2949325Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.2949757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:45.2950377Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.2950816Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.2951380Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.2951840Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.2952267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:45.2952882Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.2953317Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.2953893Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.2954351Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.2954761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:45.2955377Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.2955812Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.2956388Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.2956844Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.2957272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:45.2957740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:45.2958273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:45.2958751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T02:51:45.2959223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T02:51:45.2959877Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:45.2960557Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:45.2961240Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:45.2961917Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:45.2962470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:51:45.2962930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3 2022-11-23T02:51:45.2963405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:51:45.2963877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-11-23T02:51:45.2964526Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T02:51:45.2965204Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T02:51:45.2965880Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T02:51:45.2966557Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T02:51:45.2966941Z ok (5.334s) 2022-11-23T02:51:45.2967080Z 2022-11-23T02:51:45.2967337Z ---------------------------------------------------------------------- 2022-11-23T02:51:45.2967657Z Ran 1 test in 5.335s 2022-11-23T02:51:45.2967898Z 2022-11-23T02:51:45.2967986Z OK 2022-11-23T02:51:45.2968112Z 2022-11-23T02:51:45.2968233Z Generating XML reports... 2022-11-23T02:51:45.2968858Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123025015.xml 2022-11-23T02:51:45.2969644Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-11-23T02:51:45.2970403Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.2970917Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.2971621Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.2972176Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.2972436Z 2022-11-23T02:51:45.2972555Z Running tests... 2022-11-23T02:51:45.2973051Z ---------------------------------------------------------------------- 2022-11-23T02:51:45.2973698Z test_collectives_op_mismatch (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94282 2022-11-23T02:51:45.2974341Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94283 2022-11-23T02:51:45.2974869Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 94284 2022-11-23T02:51:45.2975383Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 94285 2022-11-23T02:51:45.2976129Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.2976776Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.2977474Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.2978023Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.2978526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:45.2979204Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.2979629Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.2980205Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.2980663Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.2981144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:45.2981775Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.2982217Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.2982799Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.2983239Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.2983670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:45.2984287Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.2984724Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.2985299Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.2985763Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.2986187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:45.2986646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:45.2987124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T02:51:45.2987602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T02:51:45.2988256Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:45.2988766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:45.2989414Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:45.2990100Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:45.2990776Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:45.2991139Z ok (5.418s) 2022-11-23T02:51:45.2991281Z 2022-11-23T02:51:45.2991552Z ---------------------------------------------------------------------- 2022-11-23T02:51:45.2991871Z Ran 1 test in 5.418s 2022-11-23T02:51:45.2992024Z 2022-11-23T02:51:45.2992110Z OK 2022-11-23T02:51:45.2992235Z 2022-11-23T02:51:45.2992355Z Generating XML reports... 2022-11-23T02:51:45.2992972Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123025024.xml 2022-11-23T02:51:45.2993632Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-11-23T02:51:45.2994248Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.2994759Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.2995346Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.2995801Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.2996018Z 2022-11-23T02:51:45.2996120Z Running tests... 2022-11-23T02:51:45.2996526Z ---------------------------------------------------------------------- 2022-11-23T02:51:45.2997065Z test_collectives_op_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94653 2022-11-23T02:51:45.2997610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94654 2022-11-23T02:51:45.2998044Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 94655 2022-11-23T02:51:45.2998540Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 94656 2022-11-23T02:51:45.2999154Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.2999594Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.3000175Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.3000633Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.3001059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:45.3001668Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.3002107Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.3002689Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.3003147Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.3003575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:45.3004197Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.3004637Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.3005201Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.3005661Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.3006088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:45.3006700Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.3007139Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.3007851Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.3008314Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.3008727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:45.3009168Z skip: Need at least 4 CUDA devices (4.739s) 2022-11-23T02:51:45.3009382Z 2022-11-23T02:51:45.3009713Z ---------------------------------------------------------------------- 2022-11-23T02:51:45.3010094Z Ran 1 test in 4.739s 2022-11-23T02:51:45.3010279Z 2022-11-23T02:51:45.3010394Z OK (skipped=1) 2022-11-23T02:51:45.3010568Z 2022-11-23T02:51:45.3010708Z Generating XML reports... 2022-11-23T02:51:45.3011456Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123025033.xml 2022-11-23T02:51:45.3012329Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-11-23T02:51:45.3013077Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.3013599Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.3014295Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.3014840Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.3015103Z 2022-11-23T02:51:45.3015230Z Running tests... 2022-11-23T02:51:45.3015723Z ---------------------------------------------------------------------- 2022-11-23T02:51:45.3016371Z test_collectives_op_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94988 2022-11-23T02:51:45.3017119Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94989 2022-11-23T02:51:45.3017648Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 94990 2022-11-23T02:51:45.3018186Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 94991 2022-11-23T02:51:45.3018920Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.3019432Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.3020011Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.3020470Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.3020887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:45.3021510Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.3021952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.3022538Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.3022994Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.3023420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:45.3024038Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.3024461Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.3025040Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.3025494Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.3025926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:45.3026543Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.3026997Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.3027581Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.3028037Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.3028465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:45.3028850Z skip: Need at least 4 CUDA devices (5.420s) 2022-11-23T02:51:45.3029029Z 2022-11-23T02:51:45.3029288Z ---------------------------------------------------------------------- 2022-11-23T02:51:45.3029605Z Ran 1 test in 5.421s 2022-11-23T02:51:45.3029759Z 2022-11-23T02:51:45.3029859Z OK (skipped=1) 2022-11-23T02:51:45.3030066Z 2022-11-23T02:51:45.3030186Z Generating XML reports... 2022-11-23T02:51:45.3030809Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123025042.xml 2022-11-23T02:51:45.3031469Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-11-23T02:51:45.3032091Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.3032514Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.3033094Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.3033549Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.3033765Z 2022-11-23T02:51:45.3033868Z Running tests... 2022-11-23T02:51:45.3034277Z ---------------------------------------------------------------------- 2022-11-23T02:51:45.3034880Z test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95323 2022-11-23T02:51:45.3035437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95324 2022-11-23T02:51:45.3035879Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 95325 2022-11-23T02:51:45.3036307Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 95326 2022-11-23T02:51:45.3036920Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.3037360Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.3037936Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.3038394Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.3038830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T02:51:45.3039448Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.3039871Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.3040448Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.3040905Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.3041332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:45.3041945Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.3042385Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.3042966Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.3043412Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.3043841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:45.3044457Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.3044895Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.3045474Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.3045934Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.3046361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T02:51:45.3046823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:45.3047359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T02:51:45.3047884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:45.3048546Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:45.3049063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T02:51:45.3049829Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:45.3050651Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:45.3051464Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:45.3052149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:51:45.3052707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:51:45.3053279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2 2022-11-23T02:51:45.3053843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3 2022-11-23T02:51:45.3054623Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T02:51:45.3055435Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T02:51:45.3056246Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T02:51:45.3057056Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes. 2022-11-23T02:51:45.3057506Z ok (5.024s) 2022-11-23T02:51:45.3057662Z 2022-11-23T02:51:45.3057988Z ---------------------------------------------------------------------- 2022-11-23T02:51:45.3058370Z Ran 1 test in 5.025s 2022-11-23T02:51:45.3058553Z 2022-11-23T02:51:45.3058654Z OK 2022-11-23T02:51:45.3058805Z 2022-11-23T02:51:45.3058949Z Generating XML reports... 2022-11-23T02:51:45.3059603Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123025051.xml 2022-11-23T02:51:45.3060266Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-11-23T02:51:45.3060886Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.3061312Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.3061895Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.3062355Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.3062573Z 2022-11-23T02:51:45.3062676Z Running tests... 2022-11-23T02:51:45.3063094Z ---------------------------------------------------------------------- 2022-11-23T02:51:45.3063619Z test_collective_hang (__main__.ProcessGroupNCCLWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95706 2022-11-23T02:51:45.3064150Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95707 2022-11-23T02:51:45.3064747Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.3065184Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.3065763Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.3066291Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.3066724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:45.3067199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:45.3067828Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.3068258Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.3068840Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.3069305Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.3069731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:45.3070252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:45.3070911Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:45.3071594Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:45.3072093Z [E ProcessGroupGloo.cpp:2802] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms 2022-11-23T02:51:45.3072531Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 2000 ms 2022-11-23T02:51:45.3072866Z ok (4.518s) 2022-11-23T02:51:45.3073009Z 2022-11-23T02:51:45.3073278Z ---------------------------------------------------------------------- 2022-11-23T02:51:45.3073601Z Ran 1 test in 4.518s 2022-11-23T02:51:45.3073756Z 2022-11-23T02:51:45.3073841Z OK 2022-11-23T02:51:45.3073966Z 2022-11-23T02:51:45.3074082Z Generating XML reports... 2022-11-23T02:51:45.3074689Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221123025100.xml 2022-11-23T02:51:45.3075352Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-11-23T02:51:45.3075973Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.3076413Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.3076991Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.3077451Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.3077670Z 2022-11-23T02:51:45.3077772Z Running tests... 2022-11-23T02:51:45.3078185Z ---------------------------------------------------------------------- 2022-11-23T02:51:45.3078717Z test_collective_shape_mismatch (__main__.ProcessGroupNCCLWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95917 2022-11-23T02:51:45.3079264Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95918 2022-11-23T02:51:45.3079874Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.3080317Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.3080896Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.3081355Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.3081790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:45.3082249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:45.3082883Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.3083390Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.3083977Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.3084438Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.3084864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:45.3085336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:45.3085990Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:45.3086655Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:45.3087219Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:51:45.3087831Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:51:45.3088161Z ok (5.108s) 2022-11-23T02:51:45.3088302Z 2022-11-23T02:51:45.3088577Z ---------------------------------------------------------------------- 2022-11-23T02:51:45.3088892Z Ran 1 test in 5.108s 2022-11-23T02:51:45.3089047Z 2022-11-23T02:51:45.3089135Z OK 2022-11-23T02:51:45.3089248Z 2022-11-23T02:51:45.3089369Z Generating XML reports... 2022-11-23T02:51:45.3090128Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221123025108.xml 2022-11-23T02:51:45.3090928Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-11-23T02:51:45.3091671Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.3092204Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.3092907Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.3093460Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.3093707Z 2022-11-23T02:51:45.3093836Z Running tests... 2022-11-23T02:51:45.3094331Z ---------------------------------------------------------------------- 2022-11-23T02:51:45.3094990Z test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96142 2022-11-23T02:51:45.3095672Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96143 2022-11-23T02:51:45.3096409Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.3096938Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.3097649Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.3098187Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.3098703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:45.3099401Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.3099839Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.3100421Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.3100880Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.3101307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:45.3101766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:45.3102314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:45.3102974Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:45.3103650Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:45.3104166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:51:45.3104640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:51:45.3105283Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:51:45.3105960Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:51:45.3106609Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:51:45.3107101Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:51:45.3107418Z ok (5.010s) 2022-11-23T02:51:45.3107560Z 2022-11-23T02:51:45.3107830Z ---------------------------------------------------------------------- 2022-11-23T02:51:45.3108148Z Ran 1 test in 5.010s 2022-11-23T02:51:45.3108300Z 2022-11-23T02:51:45.3108387Z OK 2022-11-23T02:51:45.3108513Z 2022-11-23T02:51:45.3108631Z Generating XML reports... 2022-11-23T02:51:45.3109234Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221123025117.xml 2022-11-23T02:51:45.3109894Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-11-23T02:51:45.3110521Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.3110971Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.3111552Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.3112006Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.3112225Z 2022-11-23T02:51:45.3112332Z Running tests... 2022-11-23T02:51:45.3112734Z ---------------------------------------------------------------------- 2022-11-23T02:51:45.3113264Z test_collectives_op_mismatch (__main__.ProcessGroupNCCLWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96377 2022-11-23T02:51:45.3113810Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96378 2022-11-23T02:51:45.3114423Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.3114864Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.3115449Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.3115909Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.3116338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:45.3116798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:45.3117422Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.3117860Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.3118441Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.3118896Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.3119335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:45.3119881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:45.3120531Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:45.3121210Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:45.3121773Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:51:45.3122274Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:51:45.3122591Z ok (5.276s) 2022-11-23T02:51:45.3122738Z 2022-11-23T02:51:45.3123006Z ---------------------------------------------------------------------- 2022-11-23T02:51:45.3123323Z Ran 1 test in 5.276s 2022-11-23T02:51:45.3123477Z 2022-11-23T02:51:45.3123551Z OK 2022-11-23T02:51:45.3123676Z 2022-11-23T02:51:45.3123798Z Generating XML reports... 2022-11-23T02:51:45.3124470Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221123025126.xml 2022-11-23T02:51:45.3125140Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper 2022-11-23T02:51:45.3125761Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.3126200Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.3126786Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.3127233Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.3127453Z 2022-11-23T02:51:45.3127556Z Running tests... 2022-11-23T02:51:45.3128080Z ---------------------------------------------------------------------- 2022-11-23T02:51:45.3128643Z test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96602 2022-11-23T02:51:45.3129205Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96603 2022-11-23T02:51:45.3129816Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.3130256Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.3130842Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.3131287Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.3131720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:51:45.3132337Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:51:45.3132785Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:51:45.3133371Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:51:45.3133828Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:51:45.3134257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:51:45.3134718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:45.3135376Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:45.3135889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:45.3136540Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:51:45.3137135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:51:45.3137607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:51:45.3138257Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:51:45.3138937Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:51:45.3139482Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:51:45.3139988Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:51:45.3140306Z ok (5.112s) 2022-11-23T02:51:45.3140448Z 2022-11-23T02:51:45.3140717Z ---------------------------------------------------------------------- 2022-11-23T02:51:45.3141038Z Ran 1 test in 5.113s 2022-11-23T02:51:45.3141192Z 2022-11-23T02:51:45.3141284Z OK 2022-11-23T02:51:45.3141464Z 2022-11-23T02:51:45.3141588Z Generating XML reports... 2022-11-23T02:51:45.3142200Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221123025135.xml 2022-11-23T02:51:45.3142554Z 2022-11-23T02:51:45.3145190Z ##[endgroup] 2022-11-23T02:51:45.3145823Z FINISHED PRINTING LOG FILE of distributed/test_pg_wrapper (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_pg_wrapper_7px1l7zb) 2022-11-23T02:51:45.3146143Z 2022-11-23T02:51:45.3146415Z Running distributed/test_multi_threaded_pg ... [2022-11-23 02:51:45.277214] 2022-11-23T02:51:45.3147110Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_multi_threaded_pg.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:51:45.277914] 2022-11-23T02:51:49.9231254Z 2022-11-23T02:51:49.9232268Z Expand the folded group to see the log file of distributed/test_multi_threaded_pg 2022-11-23T02:51:49.9234524Z ##[group]PRINTING LOG FILE of distributed/test_multi_threaded_pg (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_multi_threaded_pg_p93c31__) 2022-11-23T02:51:49.9236883Z Test results will be stored in test-reports/python-unittest/distributed.test_multi_threaded_pg 2022-11-23T02:51:49.9238005Z 2022-11-23T02:51:49.9238348Z Running tests... 2022-11-23T02:51:49.9239743Z ---------------------------------------------------------------------- 2022-11-23T02:51:49.9241460Z test_allgather (__main__.TestCollectivesWithBaseClass) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:49.9242968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:49.9244483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T02:51:49.9246015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T02:51:49.9248677Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:49.9250553Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:49.9252390Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:49.9254222Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:49.9255204Z ok (0.616s) 2022-11-23T02:51:49.9256386Z test_broadcast (__main__.TestCollectivesWithBaseClass) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:49.9257857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:49.9259118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T02:51:49.9260894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T02:51:49.9262662Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:49.9264487Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:49.9266317Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:49.9268135Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:49.9269114Z ok (0.022s) 2022-11-23T02:51:49.9270341Z test_broadcast_object_list (__main__.TestCollectivesWithBaseClass) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:49.9272075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:49.9273339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T02:51:49.9274577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T02:51:49.9276316Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:49.9278140Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:49.9279954Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:49.9281731Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:49.9282812Z 2 -> 4 2022-11-23T02:51:49.9283455Z 3 -> 4 2022-11-23T02:51:49.9284091Z 0 -> 4 2022-11-23T02:51:49.9284716Z 1 -> 4 2022-11-23T02:51:49.9285262Z ok (0.020s) 2022-11-23T02:51:49.9286457Z test_reduce_scatter (__main__.TestCollectivesWithBaseClass) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:49.9288044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:49.9289293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T02:51:49.9291031Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:49.9292367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T02:51:49.9294094Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:49.9295931Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:49.9297733Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:49.9298709Z ok (0.017s) 2022-11-23T02:51:49.9299880Z test_scatter (__main__.TestCollectivesWithBaseClass) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:49.9301340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:49.9302582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T02:51:49.9304304Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:49.9305652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T02:51:49.9307575Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:49.9309361Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:49.9311163Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:49.9312155Z ok (0.017s) 2022-11-23T02:51:49.9313369Z test_broadcast_object_list (__main__.TestCollectivesWithWrapper) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:51:49.9314885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:51:49.9316135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T02:51:49.9317544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T02:51:49.9319308Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:49.9321095Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:49.9322932Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:49.9324732Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T02:51:49.9325723Z ok (0.019s) 2022-11-23T02:51:49.9326088Z 2022-11-23T02:51:49.9326830Z ---------------------------------------------------------------------- 2022-11-23T02:51:49.9327673Z Ran 6 tests in 0.712s 2022-11-23T02:51:49.9328376Z 2022-11-23T02:51:49.9328615Z OK 2022-11-23T02:51:49.9328949Z 2022-11-23T02:51:49.9329255Z Generating XML reports... 2022-11-23T02:51:49.9330967Z Generated XML report: test-reports/python-unittest/distributed.test_multi_threaded_pg/TEST-TestCollectivesWithBaseClass-20221123025147.xml 2022-11-23T02:51:49.9333179Z Generated XML report: test-reports/python-unittest/distributed.test_multi_threaded_pg/TEST-TestCollectivesWithWrapper-20221123025147.xml 2022-11-23T02:51:49.9334117Z 2022-11-23T02:51:49.9334936Z ##[endgroup] 2022-11-23T02:51:49.9336562Z FINISHED PRINTING LOG FILE of distributed/test_multi_threaded_pg (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_multi_threaded_pg_p93c31__) 2022-11-23T02:51:49.9337086Z 2022-11-23T02:51:49.9337428Z Running distributed/test_dynamo_distributed ... [2022-11-23 02:51:49.923521] 2022-11-23T02:51:49.9338140Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_dynamo_distributed.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:51:49.924138] 2022-11-23T02:51:54.2142615Z 2022-11-23T02:51:54.2144202Z Expand the folded group to see the log file of distributed/test_dynamo_distributed 2022-11-23T02:51:54.2147255Z ##[group]PRINTING LOG FILE of distributed/test_dynamo_distributed (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_dynamo_distributed_f4_kj1t4) 2022-11-23T02:51:54.2148706Z 2022-11-23T02:51:54.2149707Z ##[endgroup] 2022-11-23T02:51:54.2158113Z FINISHED PRINTING LOG FILE of distributed/test_dynamo_distributed (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_dynamo_distributed_f4_kj1t4) 2022-11-23T02:51:54.2159152Z 2022-11-23T02:51:54.2160276Z Running distributed/test_c10d_spawn_ucc ... [2022-11-23 02:51:54.214811] 2022-11-23T02:51:54.2163027Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_spawn_ucc.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:51:54.215802] 2022-11-23T02:52:39.2444643Z 2022-11-23T02:52:39.2445872Z Expand the folded group to see the log file of distributed/test_c10d_spawn_ucc 2022-11-23T02:52:39.2448923Z ##[group]PRINTING LOG FILE of distributed/test_c10d_spawn_ucc (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_c10d_spawn_ucc_evkbz6z9) 2022-11-23T02:52:39.2451118Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4nfn23p0 2022-11-23T02:52:39.2452884Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4nfn23p0/_remote_module_non_scriptable.py 2022-11-23T02:52:39.2455281Z , <__main__.ProcessGroupShareTensorTest testMethod=test_shared_allreduce_ucc>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_broadcast_ucc>]> 2022-11-23T02:52:39.2457581Z test_shared_allgather_ucc (__main__.ProcessGroupShareTensorTest) 2022-11-23T02:52:39.2459026Z test_shared_allreduce_ucc (__main__.ProcessGroupShareTensorTest) 2022-11-23T02:52:39.2461102Z test_shared_broadcast_ucc (__main__.ProcessGroupShareTensorTest) 2022-11-23T02:52:39.2462319Z 2022-11-23T02:52:39.2463328Z 2022-11-23T02:52:39.2466408Z , <__main__.TestDistributedNNFunctionsUcc testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsUcc testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsUcc testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsUcc testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsUcc testMethod=test_reduce>]> 2022-11-23T02:52:39.2469132Z test_all_gather (__main__.TestDistributedNNFunctionsUcc) 2022-11-23T02:52:39.2470252Z test_all_to_all (__main__.TestDistributedNNFunctionsUcc) 2022-11-23T02:52:39.2471406Z test_all_to_all_single (__main__.TestDistributedNNFunctionsUcc) 2022-11-23T02:52:39.2472527Z test_allreduce (__main__.TestDistributedNNFunctionsUcc) 2022-11-23T02:52:39.2473671Z test_broadcast (__main__.TestDistributedNNFunctionsUcc) 2022-11-23T02:52:39.2474756Z test_reduce (__main__.TestDistributedNNFunctionsUcc) 2022-11-23T02:52:39.2476791Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:39.2478108Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:39.2479861Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:39.2481213Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:39.2482538Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptp6wkhps 2022-11-23T02:52:39.2484068Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptp6wkhps/_remote_module_non_scriptable.py 2022-11-23T02:52:39.2485868Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc 2022-11-23T02:52:39.2486663Z 2022-11-23T02:52:39.2486971Z Running tests... 2022-11-23T02:52:39.2488302Z ---------------------------------------------------------------------- 2022-11-23T02:52:39.2489535Z test_shared_allgather_ucc (__main__.ProcessGroupShareTensorTest) ... skip: UCC needed (0.001s) 2022-11-23T02:52:39.2490224Z 2022-11-23T02:52:39.2490967Z ---------------------------------------------------------------------- 2022-11-23T02:52:39.2491797Z Ran 1 test in 0.001s 2022-11-23T02:52:39.2492194Z 2022-11-23T02:52:39.2492456Z OK (skipped=1) 2022-11-23T02:52:39.2492845Z 2022-11-23T02:52:39.2493153Z Generating XML reports... 2022-11-23T02:52:39.2494854Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-ProcessGroupShareTensorTest-20221123025158.xml 2022-11-23T02:52:39.2510645Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:39.2512814Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:39.2515489Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:39.2516994Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:39.2518624Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpctjw_8cn 2022-11-23T02:52:39.2520289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpctjw_8cn/_remote_module_non_scriptable.py 2022-11-23T02:52:39.2522289Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc 2022-11-23T02:52:39.2523045Z 2022-11-23T02:52:39.2523338Z Running tests... 2022-11-23T02:52:39.2524554Z ---------------------------------------------------------------------- 2022-11-23T02:52:39.2525837Z test_shared_allreduce_ucc (__main__.ProcessGroupShareTensorTest) ... skip: UCC needed (0.001s) 2022-11-23T02:52:39.2526654Z 2022-11-23T02:52:39.2527488Z ---------------------------------------------------------------------- 2022-11-23T02:52:39.2529027Z Ran 1 test in 0.001s 2022-11-23T02:52:39.2529463Z 2022-11-23T02:52:39.2529729Z OK (skipped=1) 2022-11-23T02:52:39.2530121Z 2022-11-23T02:52:39.2530522Z Generating XML reports... 2022-11-23T02:52:39.2553832Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-ProcessGroupShareTensorTest-20221123025203.xml 2022-11-23T02:52:39.2557206Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:39.2558854Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:39.2560831Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:39.2562282Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:39.2563603Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8cir_cso 2022-11-23T02:52:39.2565144Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8cir_cso/_remote_module_non_scriptable.py 2022-11-23T02:52:39.2566922Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc 2022-11-23T02:52:39.2567875Z 2022-11-23T02:52:39.2568148Z Running tests... 2022-11-23T02:52:39.2569256Z ---------------------------------------------------------------------- 2022-11-23T02:52:39.2570468Z test_shared_broadcast_ucc (__main__.ProcessGroupShareTensorTest) ... skip: UCC needed (0.001s) 2022-11-23T02:52:39.2571167Z 2022-11-23T02:52:39.2571890Z ---------------------------------------------------------------------- 2022-11-23T02:52:39.2572711Z Ran 1 test in 0.001s 2022-11-23T02:52:39.2573159Z 2022-11-23T02:52:39.2573405Z OK (skipped=1) 2022-11-23T02:52:39.2573787Z 2022-11-23T02:52:39.2574064Z Generating XML reports... 2022-11-23T02:52:39.2575798Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-ProcessGroupShareTensorTest-20221123025207.xml 2022-11-23T02:52:39.2577784Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:39.2578927Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:39.2580447Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:39.2581645Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:39.2582807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp64fqv26w 2022-11-23T02:52:39.2584715Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp64fqv26w/_remote_module_non_scriptable.py 2022-11-23T02:52:39.2586335Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc 2022-11-23T02:52:39.2587012Z 2022-11-23T02:52:39.2587249Z Running tests... 2022-11-23T02:52:39.2588317Z ---------------------------------------------------------------------- 2022-11-23T02:52:39.2590190Z test_all_gather (__main__.TestDistributedNNFunctionsUcc) ... skip: c10d was not compiled with the UCC backend (0.000s) 2022-11-23T02:52:39.2590941Z 2022-11-23T02:52:39.2591666Z ---------------------------------------------------------------------- 2022-11-23T02:52:39.2592482Z Ran 1 test in 0.001s 2022-11-23T02:52:39.2592885Z 2022-11-23T02:52:39.2593147Z OK (skipped=1) 2022-11-23T02:52:39.2593526Z 2022-11-23T02:52:39.2593821Z Generating XML reports... 2022-11-23T02:52:39.2595465Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20221123025212.xml 2022-11-23T02:52:39.2597476Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:39.2598642Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:39.2600375Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:39.2601647Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:39.2602825Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuk515u_o 2022-11-23T02:52:39.2604172Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuk515u_o/_remote_module_non_scriptable.py 2022-11-23T02:52:39.2605760Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc 2022-11-23T02:52:39.2606436Z 2022-11-23T02:52:39.2606673Z Running tests... 2022-11-23T02:52:39.2607825Z ---------------------------------------------------------------------- 2022-11-23T02:52:39.2609075Z test_all_to_all (__main__.TestDistributedNNFunctionsUcc) ... skip: c10d was not compiled with the UCC backend (0.000s) 2022-11-23T02:52:39.2609822Z 2022-11-23T02:52:39.2610528Z ---------------------------------------------------------------------- 2022-11-23T02:52:39.2611352Z Ran 1 test in 0.001s 2022-11-23T02:52:39.2611763Z 2022-11-23T02:52:39.2612010Z OK (skipped=1) 2022-11-23T02:52:39.2612387Z 2022-11-23T02:52:39.2612686Z Generating XML reports... 2022-11-23T02:52:39.2614329Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20221123025216.xml 2022-11-23T02:52:39.2616256Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:39.2617412Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:39.2618951Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:39.2620162Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:39.2621450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp668x4pop 2022-11-23T02:52:39.2622856Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp668x4pop/_remote_module_non_scriptable.py 2022-11-23T02:52:39.2624441Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc 2022-11-23T02:52:39.2625116Z 2022-11-23T02:52:39.2625372Z Running tests... 2022-11-23T02:52:39.2626422Z ---------------------------------------------------------------------- 2022-11-23T02:52:39.2627667Z test_all_to_all_single (__main__.TestDistributedNNFunctionsUcc) ... skip: c10d was not compiled with the UCC backend (0.000s) 2022-11-23T02:52:39.2628432Z 2022-11-23T02:52:39.2629135Z ---------------------------------------------------------------------- 2022-11-23T02:52:39.2629957Z Ran 1 test in 0.001s 2022-11-23T02:52:39.2630346Z 2022-11-23T02:52:39.2630591Z OK (skipped=1) 2022-11-23T02:52:39.2630964Z 2022-11-23T02:52:39.2631260Z Generating XML reports... 2022-11-23T02:52:39.2632910Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20221123025221.xml 2022-11-23T02:52:39.2635057Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:39.2636237Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:39.2637798Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:39.2639087Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:39.2640300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4bwzfbkt 2022-11-23T02:52:39.2641722Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4bwzfbkt/_remote_module_non_scriptable.py 2022-11-23T02:52:39.2643329Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc 2022-11-23T02:52:39.2644034Z 2022-11-23T02:52:39.2644288Z Running tests... 2022-11-23T02:52:39.2645457Z ---------------------------------------------------------------------- 2022-11-23T02:52:39.2646861Z test_allreduce (__main__.TestDistributedNNFunctionsUcc) ... skip: c10d was not compiled with the UCC backend (0.001s) 2022-11-23T02:52:39.2647633Z 2022-11-23T02:52:39.2648704Z ---------------------------------------------------------------------- 2022-11-23T02:52:39.2649563Z Ran 1 test in 0.001s 2022-11-23T02:52:39.2649960Z 2022-11-23T02:52:39.2650207Z OK (skipped=1) 2022-11-23T02:52:39.2650581Z 2022-11-23T02:52:39.2650878Z Generating XML reports... 2022-11-23T02:52:39.2652535Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20221123025225.xml 2022-11-23T02:52:39.2654457Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:39.2655611Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:39.2657122Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:39.2658341Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:39.2659508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfsf_m6jd 2022-11-23T02:52:39.2660850Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfsf_m6jd/_remote_module_non_scriptable.py 2022-11-23T02:52:39.2662430Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc 2022-11-23T02:52:39.2663091Z 2022-11-23T02:52:39.2663346Z Running tests... 2022-11-23T02:52:39.2664091Z ---------------------------------------------------------------------- 2022-11-23T02:52:39.2664638Z test_broadcast (__main__.TestDistributedNNFunctionsUcc) ... skip: c10d was not compiled with the UCC backend (0.000s) 2022-11-23T02:52:39.2664920Z 2022-11-23T02:52:39.2665183Z ---------------------------------------------------------------------- 2022-11-23T02:52:39.2665497Z Ran 1 test in 0.001s 2022-11-23T02:52:39.2665648Z 2022-11-23T02:52:39.2665752Z OK (skipped=1) 2022-11-23T02:52:39.2665897Z 2022-11-23T02:52:39.2666011Z Generating XML reports... 2022-11-23T02:52:39.2666632Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20221123025230.xml 2022-11-23T02:52:39.2667347Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:52:39.2667773Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:52:39.2668350Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:52:39.2668809Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:52:39.2669251Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvx_v9fut 2022-11-23T02:52:39.2669766Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvx_v9fut/_remote_module_non_scriptable.py 2022-11-23T02:52:39.2670455Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc 2022-11-23T02:52:39.2670709Z 2022-11-23T02:52:39.2670809Z Running tests... 2022-11-23T02:52:39.2671201Z ---------------------------------------------------------------------- 2022-11-23T02:52:39.2671664Z test_reduce (__main__.TestDistributedNNFunctionsUcc) ... skip: c10d was not compiled with the UCC backend (0.000s) 2022-11-23T02:52:39.2671945Z 2022-11-23T02:52:39.2672210Z ---------------------------------------------------------------------- 2022-11-23T02:52:39.2672525Z Ran 1 test in 0.001s 2022-11-23T02:52:39.2672676Z 2022-11-23T02:52:39.2672772Z OK (skipped=1) 2022-11-23T02:52:39.2672915Z 2022-11-23T02:52:39.2673033Z Generating XML reports... 2022-11-23T02:52:39.2673653Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20221123025234.xml 2022-11-23T02:52:39.2674005Z 2022-11-23T02:52:39.2674542Z ##[endgroup] 2022-11-23T02:52:39.2675188Z FINISHED PRINTING LOG FILE of distributed/test_c10d_spawn_ucc (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_c10d_spawn_ucc_evkbz6z9) 2022-11-23T02:52:39.2675514Z 2022-11-23T02:52:39.2675783Z Running distributed/test_c10d_spawn_gloo ... [2022-11-23 02:52:39.245087] 2022-11-23T02:52:39.2676503Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_spawn_gloo.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:52:39.245740] 2022-11-23T02:55:01.8334650Z 2022-11-23T02:55:01.8335508Z Expand the folded group to see the log file of distributed/test_c10d_spawn_gloo 2022-11-23T02:55:01.8338174Z ##[group]PRINTING LOG FILE of distributed/test_c10d_spawn_gloo (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_c10d_spawn_gloo_2vixjqz0) 2022-11-23T02:55:01.8340220Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4mvte7za 2022-11-23T02:55:01.8341987Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4mvte7za/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8345106Z , <__main__.DistributedDataParallelSingleProcessTest testMethod=test_cuda>, <__main__.DistributedDataParallelSingleProcessTest testMethod=test_rnn>]> 2022-11-23T02:55:01.8347522Z test_cpu (__main__.DistributedDataParallelSingleProcessTest) 2022-11-23T02:55:01.8348701Z test_cuda (__main__.DistributedDataParallelSingleProcessTest) 2022-11-23T02:55:01.8349823Z test_rnn (__main__.DistributedDataParallelSingleProcessTest) 2022-11-23T02:55:01.8352069Z , <__main__.ProcessGroupShareTensorTest testMethod=test_shared_allgather_gloo>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_allreduce_gloo>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_broadcast_gloo>]> 2022-11-23T02:55:01.8354278Z test_shared_allgather_chunk_gloo (__main__.ProcessGroupShareTensorTest) 2022-11-23T02:55:01.8355420Z test_shared_allgather_gloo (__main__.ProcessGroupShareTensorTest) 2022-11-23T02:55:01.8356533Z test_shared_allreduce_gloo (__main__.ProcessGroupShareTensorTest) 2022-11-23T02:55:01.8357639Z test_shared_broadcast_gloo (__main__.ProcessGroupShareTensorTest) 2022-11-23T02:55:01.8358613Z 2022-11-23T02:55:01.8361800Z 2022-11-23T02:55:01.8363478Z , <__main__.TestDistributedNNFunctionsGloo testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_gather>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_reduce>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_scatter>]> 2022-11-23T02:55:01.8365423Z test_all_gather (__main__.TestDistributedNNFunctionsGloo) 2022-11-23T02:55:01.8366067Z test_all_to_all (__main__.TestDistributedNNFunctionsGloo) 2022-11-23T02:55:01.8366642Z test_all_to_all_single (__main__.TestDistributedNNFunctionsGloo) 2022-11-23T02:55:01.8367156Z test_allreduce (__main__.TestDistributedNNFunctionsGloo) 2022-11-23T02:55:01.8367914Z test_broadcast (__main__.TestDistributedNNFunctionsGloo) 2022-11-23T02:55:01.8368413Z test_gather (__main__.TestDistributedNNFunctionsGloo) 2022-11-23T02:55:01.8369084Z test_reduce (__main__.TestDistributedNNFunctionsGloo) 2022-11-23T02:55:01.8369713Z test_scatter (__main__.TestDistributedNNFunctionsGloo) 2022-11-23T02:55:01.8371791Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8372655Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8373901Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8374610Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8375351Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbxvsyr95 2022-11-23T02:55:01.8376207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbxvsyr95/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8377087Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T02:55:01.8377459Z 2022-11-23T02:55:01.8377612Z Running tests... 2022-11-23T02:55:01.8378306Z ---------------------------------------------------------------------- 2022-11-23T02:55:01.8379465Z test_cpu (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:55:01.8380838Z ok (0.045s) 2022-11-23T02:55:01.8381070Z 2022-11-23T02:55:01.8381532Z ---------------------------------------------------------------------- 2022-11-23T02:55:01.8382075Z Ran 1 test in 0.046s 2022-11-23T02:55:01.8382317Z 2022-11-23T02:55:01.8382454Z OK 2022-11-23T02:55:01.8382711Z 2022-11-23T02:55:01.8382903Z Generating XML reports... 2022-11-23T02:55:01.8383908Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20221123025243.xml 2022-11-23T02:55:01.8385011Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8385790Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8386744Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8387493Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8388257Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpahc7hlbl 2022-11-23T02:55:01.8389106Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpahc7hlbl/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8390149Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T02:55:01.8390644Z 2022-11-23T02:55:01.8390822Z Running tests... 2022-11-23T02:55:01.8391588Z ---------------------------------------------------------------------- 2022-11-23T02:55:01.8392498Z test_cuda (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:55:01.8393300Z ok (1.752s) 2022-11-23T02:55:01.8393549Z 2022-11-23T02:55:01.8394012Z ---------------------------------------------------------------------- 2022-11-23T02:55:01.8394595Z Ran 1 test in 1.753s 2022-11-23T02:55:01.8394853Z 2022-11-23T02:55:01.8395012Z OK 2022-11-23T02:55:01.8395249Z 2022-11-23T02:55:01.8395724Z Generating XML reports... 2022-11-23T02:55:01.8396845Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20221123025248.xml 2022-11-23T02:55:01.8398121Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8398865Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8399875Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8400612Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8401301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb2iy8o5u 2022-11-23T02:55:01.8402015Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb2iy8o5u/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8402934Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T02:55:01.8403312Z 2022-11-23T02:55:01.8403456Z Running tests... 2022-11-23T02:55:01.8404451Z ---------------------------------------------------------------------- 2022-11-23T02:55:01.8405207Z test_rnn (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:55:01.8405843Z ok (3.566s) 2022-11-23T02:55:01.8406035Z 2022-11-23T02:55:01.8406428Z ---------------------------------------------------------------------- 2022-11-23T02:55:01.8406887Z Ran 1 test in 3.567s 2022-11-23T02:55:01.8407104Z 2022-11-23T02:55:01.8407241Z OK 2022-11-23T02:55:01.8407417Z 2022-11-23T02:55:01.8407590Z Generating XML reports... 2022-11-23T02:55:01.8408594Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20221123025255.xml 2022-11-23T02:55:01.8409657Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8410275Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8411081Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8411719Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8412330Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8uzaq_mq 2022-11-23T02:55:01.8413030Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8uzaq_mq/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8413811Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T02:55:01.8414166Z 2022-11-23T02:55:01.8414312Z Running tests... 2022-11-23T02:55:01.8414899Z ---------------------------------------------------------------------- 2022-11-23T02:55:01.8415951Z test_shared_allgather_chunk_gloo (__main__.ProcessGroupShareTensorTest) ... /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8416695Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8417484Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8418115Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8418734Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoflzrnmm 2022-11-23T02:55:01.8419405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoflzrnmm/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8420289Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8420901Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8421819Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8422453Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8423071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplbnsmb85 2022-11-23T02:55:01.8423773Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplbnsmb85/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8424262Z ok (4.719s) 2022-11-23T02:55:01.8424463Z 2022-11-23T02:55:01.8424839Z ---------------------------------------------------------------------- 2022-11-23T02:55:01.8425298Z Ran 1 test in 4.720s 2022-11-23T02:55:01.8425510Z 2022-11-23T02:55:01.8425647Z OK 2022-11-23T02:55:01.8425838Z 2022-11-23T02:55:01.8426005Z Generating XML reports... 2022-11-23T02:55:01.8426865Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20221123025303.xml 2022-11-23T02:55:01.8427925Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8428512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8429311Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8429940Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8430554Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4lbfc2d8 2022-11-23T02:55:01.8431265Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4lbfc2d8/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8432086Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T02:55:01.8432438Z 2022-11-23T02:55:01.8432589Z Running tests... 2022-11-23T02:55:01.8433143Z ---------------------------------------------------------------------- 2022-11-23T02:55:01.8434176Z test_shared_allgather_gloo (__main__.ProcessGroupShareTensorTest) ... /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8434913Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8435721Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8436356Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8436983Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwj85xsgi 2022-11-23T02:55:01.8437684Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwj85xsgi/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8438575Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8439151Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8439968Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8440609Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8441230Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbpnvuryr 2022-11-23T02:55:01.8441941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbpnvuryr/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8442465Z ok (5.659s) 2022-11-23T02:55:01.8442669Z 2022-11-23T02:55:01.8443052Z ---------------------------------------------------------------------- 2022-11-23T02:55:01.8443475Z Ran 1 test in 5.660s 2022-11-23T02:55:01.8443691Z 2022-11-23T02:55:01.8443830Z OK 2022-11-23T02:55:01.8444011Z 2022-11-23T02:55:01.8444181Z Generating XML reports... 2022-11-23T02:55:01.8445044Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20221123025313.xml 2022-11-23T02:55:01.8445984Z [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors] 2022-11-23T02:55:01.8446919Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8447531Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8448469Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8449109Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8449737Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr76kwx9z 2022-11-23T02:55:01.8450447Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr76kwx9z/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8451373Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T02:55:01.8451744Z 2022-11-23T02:55:01.8451903Z Running tests... 2022-11-23T02:55:01.8452490Z ---------------------------------------------------------------------- 2022-11-23T02:55:01.8453519Z test_shared_allreduce_gloo (__main__.ProcessGroupShareTensorTest) ... /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8454225Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8455024Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8455657Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8456280Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy37ymc6b 2022-11-23T02:55:01.8456992Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy37ymc6b/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8457883Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8458491Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8459248Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8459882Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8460509Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprjgp6xp2 2022-11-23T02:55:01.8461219Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprjgp6xp2/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8461743Z ok (5.219s) 2022-11-23T02:55:01.8461944Z 2022-11-23T02:55:01.8462261Z ---------------------------------------------------------------------- 2022-11-23T02:55:01.8462653Z Ran 1 test in 5.219s 2022-11-23T02:55:01.8462837Z 2022-11-23T02:55:01.8462920Z OK 2022-11-23T02:55:01.8463081Z 2022-11-23T02:55:01.8463231Z Generating XML reports... 2022-11-23T02:55:01.8463950Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20221123025323.xml 2022-11-23T02:55:01.8464672Z [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors] 2022-11-23T02:55:01.8465441Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8465952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8466620Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8467120Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8467639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphc5bewit 2022-11-23T02:55:01.8468371Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphc5bewit/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8469067Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T02:55:01.8469369Z 2022-11-23T02:55:01.8469497Z Running tests... 2022-11-23T02:55:01.8469984Z ---------------------------------------------------------------------- 2022-11-23T02:55:01.8470829Z test_shared_broadcast_gloo (__main__.ProcessGroupShareTensorTest) ... /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8471445Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8472079Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8472611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8473187Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0t2588kp 2022-11-23T02:55:01.8473785Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0t2588kp/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8474529Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8475033Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8475689Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8476185Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8476695Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvi5g7l4b 2022-11-23T02:55:01.8477281Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvi5g7l4b/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8477705Z ok (5.330s) 2022-11-23T02:55:01.8477881Z 2022-11-23T02:55:01.8478199Z ---------------------------------------------------------------------- 2022-11-23T02:55:01.8478577Z Ran 1 test in 5.331s 2022-11-23T02:55:01.8478762Z 2022-11-23T02:55:01.8478869Z OK 2022-11-23T02:55:01.8479025Z 2022-11-23T02:55:01.8479139Z Generating XML reports... 2022-11-23T02:55:01.8479844Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20221123025333.xml 2022-11-23T02:55:01.8480557Z [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors] 2022-11-23T02:55:01.8481323Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8481825Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8482488Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8483018Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8483532Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvnhq1ace 2022-11-23T02:55:01.8484088Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvnhq1ace/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8484771Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T02:55:01.8485067Z 2022-11-23T02:55:01.8485196Z Running tests... 2022-11-23T02:55:01.8485682Z ---------------------------------------------------------------------- 2022-11-23T02:55:01.8486283Z test_all_gather (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98810 2022-11-23T02:55:01.8486879Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98811 2022-11-23T02:55:01.8487579Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8488244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8488924Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8489524Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8490133Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7a8z3kfz 2022-11-23T02:55:01.8490823Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7a8z3kfz/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8491700Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8492296Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8493052Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8493768Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8494385Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpabvku4nl 2022-11-23T02:55:01.8495070Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpabvku4nl/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8495728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:55:01.8496357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:55:01.8496988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:55:01.8497624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:55:01.8498498Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:55:01.8499422Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:55:01.8499956Z ok (5.049s) 2022-11-23T02:55:01.8500148Z 2022-11-23T02:55:01.8500515Z ---------------------------------------------------------------------- 2022-11-23T02:55:01.8500959Z Ran 1 test in 5.049s 2022-11-23T02:55:01.8501163Z 2022-11-23T02:55:01.8501290Z OK 2022-11-23T02:55:01.8501467Z 2022-11-23T02:55:01.8501632Z Generating XML reports... 2022-11-23T02:55:01.8502435Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123025343.xml 2022-11-23T02:55:01.8503245Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8503741Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8504393Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8504916Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8505426Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptieibit3 2022-11-23T02:55:01.8506009Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptieibit3/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8506673Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T02:55:01.8506967Z 2022-11-23T02:55:01.8507098Z Running tests... 2022-11-23T02:55:01.8507579Z ---------------------------------------------------------------------- 2022-11-23T02:55:01.8508160Z test_all_to_all (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99025 2022-11-23T02:55:01.8508763Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99026 2022-11-23T02:55:01.8509460Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8510037Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8510676Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8511200Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8511705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6b18nzzk 2022-11-23T02:55:01.8512287Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6b18nzzk/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8513031Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8513539Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8514260Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8514801Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8515288Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4lw3l9m7 2022-11-23T02:55:01.8515878Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4lw3l9m7/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8516440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:55:01.8516976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:55:01.8517518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:55:01.8518069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:55:01.8518815Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:55:01.8519570Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:55:01.8520021Z ok (5.040s) 2022-11-23T02:55:01.8520190Z 2022-11-23T02:55:01.8520502Z ---------------------------------------------------------------------- 2022-11-23T02:55:01.8520892Z Ran 1 test in 5.040s 2022-11-23T02:55:01.8521074Z 2022-11-23T02:55:01.8521190Z OK 2022-11-23T02:55:01.8521348Z 2022-11-23T02:55:01.8521493Z Generating XML reports... 2022-11-23T02:55:01.8522213Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123025352.xml 2022-11-23T02:55:01.8523006Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8523518Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8524186Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8524716Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8525235Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx3jga1r7 2022-11-23T02:55:01.8525822Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx3jga1r7/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8526504Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T02:55:01.8526802Z 2022-11-23T02:55:01.8526929Z Running tests... 2022-11-23T02:55:01.8527384Z ---------------------------------------------------------------------- 2022-11-23T02:55:01.8528048Z test_all_to_all_single (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99240 2022-11-23T02:55:01.8528662Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99241 2022-11-23T02:55:01.8529536Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8530148Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8530954Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8531585Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8532165Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_cv6i4ql 2022-11-23T02:55:01.8532855Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_cv6i4ql/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8533753Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8534366Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8535238Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8535897Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8536519Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7uznuuh2 2022-11-23T02:55:01.8537190Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7uznuuh2/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8537863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:55:01.8538508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:55:01.8539153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:55:01.8540057Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:55:01.8540765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:55:01.8541669Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:55:01.8542123Z ok (5.141s) 2022-11-23T02:55:01.8542260Z 2022-11-23T02:55:01.8542576Z ---------------------------------------------------------------------- 2022-11-23T02:55:01.8542968Z Ran 1 test in 5.141s 2022-11-23T02:55:01.8543150Z 2022-11-23T02:55:01.8543263Z OK 2022-11-23T02:55:01.8543420Z 2022-11-23T02:55:01.8543563Z Generating XML reports... 2022-11-23T02:55:01.8544297Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123025402.xml 2022-11-23T02:55:01.8545125Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8545643Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8546280Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8546819Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8547352Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpilh3afm3 2022-11-23T02:55:01.8547944Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpilh3afm3/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8548638Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T02:55:01.8548936Z 2022-11-23T02:55:01.8549066Z Running tests... 2022-11-23T02:55:01.8549551Z ---------------------------------------------------------------------- 2022-11-23T02:55:01.8550117Z test_allreduce (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99455 2022-11-23T02:55:01.8550725Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99456 2022-11-23T02:55:01.8551522Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8552039Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8552708Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8553236Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8553798Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmx4k7pgi 2022-11-23T02:55:01.8554473Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmx4k7pgi/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8555364Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8555969Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8556838Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8557481Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8558108Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg498x0ng 2022-11-23T02:55:01.8558811Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg498x0ng/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8559485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:55:01.8560087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:55:01.8560729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:55:01.8561628Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:55:01.8562267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:55:01.8563010Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:55:01.8563461Z ok (5.038s) 2022-11-23T02:55:01.8563631Z 2022-11-23T02:55:01.8563938Z ---------------------------------------------------------------------- 2022-11-23T02:55:01.8564292Z Ran 1 test in 5.039s 2022-11-23T02:55:01.8564476Z 2022-11-23T02:55:01.8564590Z OK 2022-11-23T02:55:01.8564743Z 2022-11-23T02:55:01.8564888Z Generating XML reports... 2022-11-23T02:55:01.8565613Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123025411.xml 2022-11-23T02:55:01.8566437Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8566953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8567628Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8568280Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8568799Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplhim7bsu 2022-11-23T02:55:01.8569446Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplhim7bsu/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8570283Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T02:55:01.8570638Z 2022-11-23T02:55:01.8570788Z Running tests... 2022-11-23T02:55:01.8571373Z ---------------------------------------------------------------------- 2022-11-23T02:55:01.8572089Z test_broadcast (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99670 2022-11-23T02:55:01.8572820Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99671 2022-11-23T02:55:01.8573742Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8574348Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8575149Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8575784Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8576400Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuxo9uhoc 2022-11-23T02:55:01.8577096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuxo9uhoc/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8577997Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8578584Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8579462Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8580101Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8580720Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu9335nym 2022-11-23T02:55:01.8581422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu9335nym/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8582067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:55:01.8582600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:55:01.8583109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:55:01.8583656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:55:01.8584413Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:55:01.8585180Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:55:01.8585628Z ok (6.141s) 2022-11-23T02:55:01.8585800Z 2022-11-23T02:55:01.8586116Z ---------------------------------------------------------------------- 2022-11-23T02:55:01.8586503Z Ran 1 test in 6.141s 2022-11-23T02:55:01.8586686Z 2022-11-23T02:55:01.8586800Z OK 2022-11-23T02:55:01.8586924Z 2022-11-23T02:55:01.8587071Z Generating XML reports... 2022-11-23T02:55:01.8587797Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123025421.xml 2022-11-23T02:55:01.8588620Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8589129Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8589798Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8590327Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8590847Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_t_8ilcw 2022-11-23T02:55:01.8591400Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_t_8ilcw/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8592081Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T02:55:01.8592378Z 2022-11-23T02:55:01.8592506Z Running tests... 2022-11-23T02:55:01.8592989Z ---------------------------------------------------------------------- 2022-11-23T02:55:01.8593552Z test_gather (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99885 2022-11-23T02:55:01.8594174Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99886 2022-11-23T02:55:01.8594814Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8595251Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8595853Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8596338Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8596808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2zwt_sb8 2022-11-23T02:55:01.8597338Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2zwt_sb8/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8598007Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8598521Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8599147Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8599601Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8600071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpalm8jtkr 2022-11-23T02:55:01.8600614Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpalm8jtkr/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8601131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:55:01.8601613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:55:01.8602105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:55:01.8602603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:55:01.8603257Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:55:01.8603959Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:55:01.8604359Z ok (5.846s) 2022-11-23T02:55:01.8604510Z 2022-11-23T02:55:01.8604794Z ---------------------------------------------------------------------- 2022-11-23T02:55:01.8605137Z Ran 1 test in 5.847s 2022-11-23T02:55:01.8605302Z 2022-11-23T02:55:01.8605399Z OK 2022-11-23T02:55:01.8605537Z 2022-11-23T02:55:01.8605665Z Generating XML reports... 2022-11-23T02:55:01.8606297Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123025432.xml 2022-11-23T02:55:01.8607052Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8607519Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8608213Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8608699Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8609173Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxln8a4r5 2022-11-23T02:55:01.8609807Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxln8a4r5/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8610536Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T02:55:01.8610857Z 2022-11-23T02:55:01.8610987Z Running tests... 2022-11-23T02:55:01.8611504Z ---------------------------------------------------------------------- 2022-11-23T02:55:01.8612156Z test_reduce (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100100 2022-11-23T02:55:01.8612915Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100101 2022-11-23T02:55:01.8613691Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8614249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8614982Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8615537Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8616098Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpviqc60lj 2022-11-23T02:55:01.8616744Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpviqc60lj/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8617549Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8618215Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8618990Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8619566Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8620103Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxdf7egay 2022-11-23T02:55:01.8620748Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxdf7egay/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8621374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:55:01.8621946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:55:01.8622483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:55:01.8622988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:55:01.8623673Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:55:01.8624389Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:55:01.8624763Z ok (5.240s) 2022-11-23T02:55:01.8624913Z 2022-11-23T02:55:01.8625191Z ---------------------------------------------------------------------- 2022-11-23T02:55:01.8625534Z Ran 1 test in 5.241s 2022-11-23T02:55:01.8625699Z 2022-11-23T02:55:01.8625799Z OK 2022-11-23T02:55:01.8625940Z 2022-11-23T02:55:01.8626069Z Generating XML reports... 2022-11-23T02:55:01.8626731Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123025442.xml 2022-11-23T02:55:01.8627492Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8627936Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8628538Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8629020Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8629488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpey9cqof5 2022-11-23T02:55:01.8630110Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpey9cqof5/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8630742Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-11-23T02:55:01.8631015Z 2022-11-23T02:55:01.8631130Z Running tests... 2022-11-23T02:55:01.8631541Z ---------------------------------------------------------------------- 2022-11-23T02:55:01.8632081Z test_scatter (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100315 2022-11-23T02:55:01.8632703Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100316 2022-11-23T02:55:01.8633349Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8633813Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8634416Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8634901Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8635343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpczbkin29 2022-11-23T02:55:01.8635886Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpczbkin29/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8636624Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:01.8637107Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:01.8637713Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:01.8638195Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:01.8638663Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsi5_j_3_ 2022-11-23T02:55:01.8639198Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsi5_j_3_/_remote_module_non_scriptable.py 2022-11-23T02:55:01.8639686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:55:01.8640166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:55:01.8640657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:55:01.8641167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:55:01.8641850Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:55:01.8642550Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:55:01.8642955Z ok (5.140s) 2022-11-23T02:55:01.8643106Z 2022-11-23T02:55:01.8643361Z ---------------------------------------------------------------------- 2022-11-23T02:55:01.8643705Z Ran 1 test in 5.140s 2022-11-23T02:55:01.8643873Z 2022-11-23T02:55:01.8643970Z OK 2022-11-23T02:55:01.8644106Z 2022-11-23T02:55:01.8644242Z Generating XML reports... 2022-11-23T02:55:01.8644899Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123025452.xml 2022-11-23T02:55:01.8645271Z 2022-11-23T02:55:01.8645693Z ##[endgroup] 2022-11-23T02:55:01.8646286Z FINISHED PRINTING LOG FILE of distributed/test_c10d_spawn_gloo (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_c10d_spawn_gloo_2vixjqz0) 2022-11-23T02:55:01.8646625Z 2022-11-23T02:55:01.8646921Z Running distributed/test_c10d_object_collectives ... [2022-11-23 02:55:01.834793] 2022-11-23T02:55:01.8647658Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_object_collectives.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:55:01.835433] 2022-11-23T02:55:24.4907662Z 2022-11-23T02:55:24.4912250Z Expand the folded group to see the log file of distributed/test_c10d_object_collectives 2022-11-23T02:55:24.4914592Z ##[group]PRINTING LOG FILE of distributed/test_c10d_object_collectives (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_c10d_object_collectives_7m9zub2v) 2022-11-23T02:55:24.4916755Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_object_collectives 2022-11-23T02:55:24.4918428Z 2022-11-23T02:55:24.4918848Z Running tests... 2022-11-23T02:55:24.4920470Z ---------------------------------------------------------------------- 2022-11-23T02:55:24.4922279Z test_all_gather_object (__main__.TestObjectCollectives) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100530 2022-11-23T02:55:24.4924385Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100531 2022-11-23T02:55:24.4927104Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:24.4928922Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:24.4930838Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:24.4932596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:24.4934174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:55:24.4936408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:55:24.4938835Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:24.4940364Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:24.4942416Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:24.4944112Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:24.4945517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:55:24.4946949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:55:24.4948944Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:55:24.4951043Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:55:24.4952712Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:55:24.4954227Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:55:24.4955211Z ok (5.021s) 2022-11-23T02:55:24.4956534Z test_broadcast_object_list (__main__.TestObjectCollectives) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100673 2022-11-23T02:55:24.4958114Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100674 2022-11-23T02:55:24.4959964Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:24.4961299Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:24.4963016Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:24.4964407Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:24.4965704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:55:24.4967133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:55:24.4969320Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:24.4970525Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:24.4972113Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:24.4973318Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:24.4974496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:55:24.4976077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:55:24.4977869Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:55:24.4979736Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:55:24.4981287Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:55:24.4982685Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:55:24.4983518Z ok (4.730s) 2022-11-23T02:55:24.4984670Z test_gather_object (__main__.TestObjectCollectives) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100818 2022-11-23T02:55:24.4986072Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100819 2022-11-23T02:55:24.4987917Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:24.4989149Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:24.4990748Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:24.4992003Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:24.4993167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:55:24.4994412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:55:24.4996133Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:24.4997328Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:24.4998899Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:24.5000164Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:24.5001328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:55:24.5002606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:55:24.5004334Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:55:24.5006192Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:55:24.5007877Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:55:24.5009667Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:55:24.5010549Z ok (4.429s) 2022-11-23T02:55:24.5011736Z test_scatter_object_list (__main__.TestObjectCollectives) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100961 2022-11-23T02:55:24.5013168Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100962 2022-11-23T02:55:24.5014809Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:24.5016015Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:24.5017596Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:24.5018849Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:24.5020002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:55:24.5021289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:55:24.5023013Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:55:24.5024429Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:55:24.5025985Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:55:24.5027234Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:55:24.5028387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:55:24.5029665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:55:24.5031436Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:55:24.5033281Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:55:24.5034840Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:55:24.5036324Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:55:24.5037217Z ok (4.526s) 2022-11-23T02:55:24.5037603Z 2022-11-23T02:55:24.5038370Z ---------------------------------------------------------------------- 2022-11-23T02:55:24.5039266Z Ran 4 tests in 18.707s 2022-11-23T02:55:24.5039686Z 2022-11-23T02:55:24.5039927Z OK 2022-11-23T02:55:24.5040276Z 2022-11-23T02:55:24.5040605Z Generating XML reports... 2022-11-23T02:55:24.5042300Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_object_collectives/TEST-TestObjectCollectives-20221123025503.xml 2022-11-23T02:55:24.5043238Z 2022-11-23T02:55:24.5044032Z ##[endgroup] 2022-11-23T02:55:24.5045760Z FINISHED PRINTING LOG FILE of distributed/test_c10d_object_collectives (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_c10d_object_collectives_7m9zub2v) 2022-11-23T02:55:24.5046716Z 2022-11-23T02:55:24.5047408Z Running distributed/test_c10d_gloo ... [2022-11-23 02:55:24.491399] 2022-11-23T02:55:24.5049486Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_gloo.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:55:24.492061] 2022-11-23T03:14:26.9057763Z 2022-11-23T03:14:26.9063163Z Expand the folded group to see the log file of distributed/test_c10d_gloo 2022-11-23T03:14:26.9065725Z ##[group]PRINTING LOG FILE of distributed/test_c10d_gloo (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_c10d_gloo_rs8ndduz) 2022-11-23T03:14:26.9068073Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw0kk0nn0 2022-11-23T03:14:26.9069825Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw0kk0nn0/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9074128Z , <__main__.CommTest testMethod=test_broadcast_coalesced_gloo_cuda>, <__main__.CommTest testMethod=test_gloo_barrier_device_ids>, <__main__.CommTest testMethod=test_gloo_rank_membership>, <__main__.CommTest testMethod=test_gloo_warn_not_in_group>, <__main__.CommTest testMethod=test_sequence_num_incremented_gloo_default>, <__main__.CommTest testMethod=test_sequence_num_incremented_gloo_subgroup>, <__main__.CommTest testMethod=test_sequence_num_set_default_pg_gloo>, <__main__.CommTest testMethod=test_sequence_num_set_gloo_new_group>, <__main__.CommTest testMethod=test_tensor_dtype_complex>, <__main__.CommTest testMethod=test_tensor_dtype_mismatch>]> 2022-11-23T03:14:26.9078552Z test_broadcast_coalesced_gloo_cpu (__main__.CommTest) 2022-11-23T03:14:26.9079965Z test_broadcast_coalesced_gloo_cuda (__main__.CommTest) 2022-11-23T03:14:26.9081307Z test_gloo_barrier_device_ids (__main__.CommTest) 2022-11-23T03:14:26.9082538Z test_gloo_rank_membership (__main__.CommTest) 2022-11-23T03:14:26.9083615Z test_gloo_warn_not_in_group (__main__.CommTest) 2022-11-23T03:14:26.9084677Z test_sequence_num_incremented_gloo_default (__main__.CommTest) 2022-11-23T03:14:26.9086553Z test_sequence_num_incremented_gloo_subgroup (__main__.CommTest) 2022-11-23T03:14:26.9088320Z test_sequence_num_set_default_pg_gloo (__main__.CommTest) 2022-11-23T03:14:26.9089760Z test_sequence_num_set_gloo_new_group (__main__.CommTest) 2022-11-23T03:14:26.9090985Z test_tensor_dtype_complex (__main__.CommTest) 2022-11-23T03:14:26.9092148Z test_tensor_dtype_mismatch (__main__.CommTest) 2022-11-23T03:14:26.9096619Z , <__main__.CompilerTest testMethod=test_allgather_work_wait_gpu>, <__main__.CompilerTest testMethod=test_allreduce_work_wait_cpu>, <__main__.CompilerTest testMethod=test_allreduce_work_wait_gpu>, <__main__.CompilerTest testMethod=test_broadcast_work_wait_cpu>, <__main__.CompilerTest testMethod=test_broadcast_work_wait_gpu>, <__main__.CompilerTest testMethod=test_consecutive_comm_work_wait_cpu>, <__main__.CompilerTest testMethod=test_consecutive_comm_work_wait_gpu>, <__main__.CompilerTest testMethod=test_nested_comm_tensor_wrapping>, <__main__.CompilerTest testMethod=test_scatter_work_wait_cpu>, <__main__.CompilerTest testMethod=test_scatter_work_wait_gpu>]> 2022-11-23T03:14:26.9100684Z test_allgather_work_wait_cpu (__main__.CompilerTest) 2022-11-23T03:14:26.9101956Z test_allgather_work_wait_gpu (__main__.CompilerTest) 2022-11-23T03:14:26.9103236Z test_allreduce_work_wait_cpu (__main__.CompilerTest) 2022-11-23T03:14:26.9104574Z test_allreduce_work_wait_gpu (__main__.CompilerTest) 2022-11-23T03:14:26.9105913Z test_broadcast_work_wait_cpu (__main__.CompilerTest) 2022-11-23T03:14:26.9107142Z test_broadcast_work_wait_gpu (__main__.CompilerTest) 2022-11-23T03:14:26.9108520Z test_consecutive_comm_work_wait_cpu (__main__.CompilerTest) 2022-11-23T03:14:26.9109863Z test_consecutive_comm_work_wait_gpu (__main__.CompilerTest) 2022-11-23T03:14:26.9111169Z test_nested_comm_tensor_wrapping (__main__.CompilerTest) 2022-11-23T03:14:26.9112466Z test_scatter_work_wait_cpu (__main__.CompilerTest) 2022-11-23T03:14:26.9113686Z test_scatter_work_wait_gpu (__main__.CompilerTest) 2022-11-23T03:14:26.9132584Z , <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_cpu>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_gpu_gloo>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_register_just_once>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_sparse_gradients>, <__main__.DistributedDataParallelTest testMethod=test_ddp_invalid_comm_hook_init>, <__main__.DistributedDataParallelTest testMethod=test_ddp_invalid_comm_hook_return_type>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_when_unused_parameters_empty>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad_with_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad_with_static_graph>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_1gpu_module_device_ids_integer_list>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_1gpu_module_device_ids_torch_device_list>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_2gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_4gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_cpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_cpu_module_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_ignored_output>, <__main__.DistributedDataParallelTest testMethod=test_ignored_output_with_unused_parameters>, <__main__.DistributedDataParallelTest testMethod=test_ignored_sharded_tensor>, <__main__.DistributedDataParallelTest testMethod=test_invalid_powerSGD_state>, <__main__.DistributedDataParallelTest testMethod=test_save_load_checkpoint>, <__main__.DistributedDataParallelTest testMethod=test_sparse_gradients>, <__main__.DistributedDataParallelTest testMethod=test_sparse_gradients_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_empty_input>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_only_empty_input>]> 2022-11-23T03:14:26.9151247Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9152866Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9154648Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9156437Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9158234Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9160029Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9161824Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9163467Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9165136Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9166764Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9168861Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9170741Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9172509Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9174189Z test_ddp_comm_hook_future_passing_cpu (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9175047Z test_ddp_comm_hook_future_passing_gpu_gloo (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9175776Z test_ddp_comm_hook_register_just_once (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9176439Z test_ddp_comm_hook_sparse_gradients (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9177199Z test_ddp_invalid_comm_hook_init (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9177928Z test_ddp_invalid_comm_hook_return_type (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9178702Z test_find_unused_parameters_when_unused_parameters_empty (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9179521Z test_global_local_unused_params_grad (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9180470Z test_global_local_unused_params_grad_with_grad_is_view (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9181305Z test_global_local_unused_params_grad_with_static_graph (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9182043Z test_gloo_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9182848Z test_gloo_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9183641Z test_gloo_backend_2gpu_module (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9184315Z test_gloo_backend_4gpu_module (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9185035Z test_gloo_backend_cpu_module (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9185735Z test_gloo_backend_cpu_module_grad_is_view (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9186439Z test_ignored_output (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9187199Z test_ignored_output_with_unused_parameters (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9187952Z test_ignored_sharded_tensor (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9188644Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9189321Z test_save_load_checkpoint (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9190024Z test_sparse_gradients (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9190743Z test_sparse_gradients_grad_is_view (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9191538Z test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9192201Z test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9193790Z , <__main__.GlooProcessGroupWithDispatchedCollectivesTests testMethod=test_allreduce_coalesced>, <__main__.GlooProcessGroupWithDispatchedCollectivesTests testMethod=test_collectives>, <__main__.GlooProcessGroupWithDispatchedCollectivesTests testMethod=test_monitored_barrier>]> 2022-11-23T03:14:26.9195481Z test_allgather_coalesced (__main__.GlooProcessGroupWithDispatchedCollectivesTests) 2022-11-23T03:14:26.9196326Z test_allreduce_coalesced (__main__.GlooProcessGroupWithDispatchedCollectivesTests) 2022-11-23T03:14:26.9197188Z test_collectives (__main__.GlooProcessGroupWithDispatchedCollectivesTests) 2022-11-23T03:14:26.9197980Z test_monitored_barrier (__main__.GlooProcessGroupWithDispatchedCollectivesTests) 2022-11-23T03:14:26.9198704Z 2022-11-23T03:14:26.9206526Z , <__main__.ProcessGroupGlooTest testMethod=test_allgather_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_coalesced_async>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_coalesced_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_noncontiguous_input>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_cuda_using_work_api>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_using_work_api>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_async>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_basics>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_checks_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_barrier_implies_wait>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_basics>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_checks>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_stress>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_empty_tensors>, <__main__.ProcessGroupGlooTest testMethod=test_gather_basics>, <__main__.ProcessGroupGlooTest testMethod=test_gather_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_gather_checks>, <__main__.ProcessGroupGlooTest testMethod=test_gather_noncontiguous_input>, <__main__.ProcessGroupGlooTest testMethod=test_gather_stress>, <__main__.ProcessGroupGlooTest testMethod=test_gather_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_multi_device_constructor>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_checks>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_stress>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_round_robin>, <__main__.ProcessGroupGlooTest testMethod=test_round_robin_create_destroy>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_basics>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_checks>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_stress>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_send_recv_all_to_all>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_checks>]> 2022-11-23T03:14:26.9214427Z test_allgather_basics (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9215147Z test_allgather_basics_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9215771Z test_allgather_checks (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9216497Z test_allgather_coalesced_async (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9217219Z test_allgather_coalesced_checks (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9217887Z test_allgather_noncontiguous_input (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9218518Z test_allgather_stress (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9219154Z test_allgather_stress_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9219817Z test_allreduce_basics (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9220493Z test_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9221213Z test_allreduce_basics_cuda_using_work_api (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9221940Z test_allreduce_basics_using_work_api (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9222652Z test_allreduce_checks (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9223256Z test_allreduce_coalesced_async (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9223951Z test_allreduce_coalesced_basics (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9224662Z test_allreduce_coalesced_checks (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9225350Z test_allreduce_coalesced_checks_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9226053Z test_allreduce_coalesced_stress (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9226683Z test_allreduce_stress (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9227316Z test_allreduce_stress_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9228003Z test_barrier_implies_wait (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9228625Z test_broadcast_basics (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9229374Z test_broadcast_basics_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9229990Z test_broadcast_checks (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9230665Z test_broadcast_stress (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9231302Z test_broadcast_stress_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9231902Z test_empty_tensors (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9232548Z test_gather_basics (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9233188Z test_gather_basics_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9233819Z test_gather_checks (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9234424Z test_gather_noncontiguous_input (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9234930Z test_gather_stress (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9235365Z test_gather_stress_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9235853Z test_multi_device_constructor (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9236405Z test_reduce_basics (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9236867Z test_reduce_basics_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9237317Z test_reduce_checks (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9237766Z test_reduce_stress (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9238192Z test_reduce_stress_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9238646Z test_round_robin (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9239108Z test_round_robin_create_destroy (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9239571Z test_scatter_basics (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9240032Z test_scatter_basics_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9240488Z test_scatter_checks (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9240900Z test_scatter_stress (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9241367Z test_scatter_stress_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9241836Z test_send_recv_all_to_all (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9242307Z test_sparse_allreduce_basics (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9242802Z test_sparse_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9243291Z test_sparse_allreduce_checks (__main__.ProcessGroupGlooTest) 2022-11-23T03:14:26.9244336Z , <__main__.ReducerTest testMethod=test_forward_backward_optimizer>, <__main__.ReducerTest testMethod=test_forward_backward_unused_parameters>, <__main__.ReducerTest testMethod=test_multi_dtype_multi_bucket>, <__main__.ReducerTest testMethod=test_multi_dtype_single_bucket>, <__main__.ReducerTest testMethod=test_single_dtype_single_bucket>]> 2022-11-23T03:14:26.9245304Z test_forward_backward (__main__.ReducerTest) 2022-11-23T03:14:26.9245713Z test_forward_backward_optimizer (__main__.ReducerTest) 2022-11-23T03:14:26.9246169Z test_forward_backward_unused_parameters (__main__.ReducerTest) 2022-11-23T03:14:26.9246619Z test_multi_dtype_multi_bucket (__main__.ReducerTest) 2022-11-23T03:14:26.9247056Z test_multi_dtype_single_bucket (__main__.ReducerTest) 2022-11-23T03:14:26.9247492Z test_single_dtype_single_bucket (__main__.ReducerTest) 2022-11-23T03:14:26.9248132Z ]> 2022-11-23T03:14:26.9248613Z test_logging_init (__main__.RendezvousEnvTest) 2022-11-23T03:14:26.9249012Z 2022-11-23T03:14:26.9249529Z ]> 2022-11-23T03:14:26.9250055Z test_default_store_timeout_gloo (__main__.TimeoutTest) 2022-11-23T03:14:26.9250955Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9251514Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9252357Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9252902Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9253461Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkkfr2rkx 2022-11-23T03:14:26.9254113Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkkfr2rkx/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9254854Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9255169Z 2022-11-23T03:14:26.9255317Z Running tests... 2022-11-23T03:14:26.9255843Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9256465Z test_broadcast_coalesced_gloo_cpu (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101237 2022-11-23T03:14:26.9257147Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101238 2022-11-23T03:14:26.9257926Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9258480Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9259207Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9259788Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9260342Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3807up77 2022-11-23T03:14:26.9260988Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3807up77/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9261574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9262345Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9262908Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9263638Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9264211Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9264780Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp6kgud19 2022-11-23T03:14:26.9265358Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp6kgud19/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9265875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9266212Z ok (4.821s) 2022-11-23T03:14:26.9266364Z 2022-11-23T03:14:26.9266656Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9266994Z Ran 1 test in 4.821s 2022-11-23T03:14:26.9267163Z 2022-11-23T03:14:26.9267259Z OK 2022-11-23T03:14:26.9267401Z 2022-11-23T03:14:26.9267532Z Generating XML reports... 2022-11-23T03:14:26.9268107Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123025528.xml 2022-11-23T03:14:26.9268768Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9269232Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9269836Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9270323Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9270786Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_u_86t_z 2022-11-23T03:14:26.9271317Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_u_86t_z/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9271934Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9272272Z 2022-11-23T03:14:26.9272387Z Running tests... 2022-11-23T03:14:26.9272800Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9273322Z test_broadcast_coalesced_gloo_cuda (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101444 2022-11-23T03:14:26.9273857Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101445 2022-11-23T03:14:26.9274501Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9274961Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9275562Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9276045Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9276541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp554nyax 2022-11-23T03:14:26.9277089Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp554nyax/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9277597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9278244Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9278710Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9279314Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9279792Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9280232Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0i2unfhh 2022-11-23T03:14:26.9280778Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0i2unfhh/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9281298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9281653Z ok (5.426s) 2022-11-23T03:14:26.9281803Z 2022-11-23T03:14:26.9282088Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9282429Z Ran 1 test in 5.427s 2022-11-23T03:14:26.9282591Z 2022-11-23T03:14:26.9282689Z OK 2022-11-23T03:14:26.9282825Z 2022-11-23T03:14:26.9282927Z Generating XML reports... 2022-11-23T03:14:26.9283495Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123025537.xml 2022-11-23T03:14:26.9284182Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9284646Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9285255Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9285749Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9286218Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiayt4bel 2022-11-23T03:14:26.9286728Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiayt4bel/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9287349Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9287611Z 2022-11-23T03:14:26.9287861Z Running tests... 2022-11-23T03:14:26.9288319Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9288879Z test_gloo_barrier_device_ids (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101655 2022-11-23T03:14:26.9289517Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101656 2022-11-23T03:14:26.9290294Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9290978Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9291682Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9292252Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9292811Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwoo8lt_o 2022-11-23T03:14:26.9293455Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwoo8lt_o/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9294074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9294844Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9295395Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9296168Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9296754Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9297308Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppjwtctff 2022-11-23T03:14:26.9297957Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppjwtctff/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9298570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9299155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:14:26.9299981Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9300603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:14:26.9301412Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9301899Z ok (4.718s) 2022-11-23T03:14:26.9302073Z 2022-11-23T03:14:26.9302415Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9302821Z Ran 1 test in 4.719s 2022-11-23T03:14:26.9303015Z 2022-11-23T03:14:26.9303124Z OK 2022-11-23T03:14:26.9303290Z 2022-11-23T03:14:26.9303441Z Generating XML reports... 2022-11-23T03:14:26.9304094Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123025546.xml 2022-11-23T03:14:26.9304922Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9305406Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9306014Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9306498Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9306965Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa5ii7wb0 2022-11-23T03:14:26.9307499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa5ii7wb0/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9308117Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9308355Z 2022-11-23T03:14:26.9308470Z Running tests... 2022-11-23T03:14:26.9308904Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9309411Z test_gloo_rank_membership (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101862 2022-11-23T03:14:26.9309934Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101863 2022-11-23T03:14:26.9310576Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9311109Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9311717Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9312170Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9312637Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp99la79zh 2022-11-23T03:14:26.9313170Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp99la79zh/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9313683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9314326Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9314789Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9315451Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9315915Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9316382Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpein64o8t 2022-11-23T03:14:26.9316919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpein64o8t/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9317427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9317918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:14:26.9318417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:14:26.9319096Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9319643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T03:14:26.9320291Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9320821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T03:14:26.9321492Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:14:26.9322184Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:14:26.9322584Z ok (4.774s) 2022-11-23T03:14:26.9322733Z 2022-11-23T03:14:26.9323020Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9323370Z Ran 1 test in 4.774s 2022-11-23T03:14:26.9323534Z 2022-11-23T03:14:26.9323606Z OK 2022-11-23T03:14:26.9323744Z 2022-11-23T03:14:26.9323872Z Generating XML reports... 2022-11-23T03:14:26.9324437Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123025555.xml 2022-11-23T03:14:26.9325123Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9325587Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9326189Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9326666Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9327107Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpojdpy95i 2022-11-23T03:14:26.9327646Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpojdpy95i/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9328359Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9328703Z 2022-11-23T03:14:26.9328844Z Running tests... 2022-11-23T03:14:26.9329367Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9329972Z test_gloo_warn_not_in_group (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102072 2022-11-23T03:14:26.9330594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102073 2022-11-23T03:14:26.9331374Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9331902Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9332628Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9333206Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9333840Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgub86olp 2022-11-23T03:14:26.9334514Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgub86olp/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9335124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9335906Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9336424Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9337146Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9337713Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9338271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps1wz__qs 2022-11-23T03:14:26.9338926Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps1wz__qs/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9339538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9340122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:14:26.9340905Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9341552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:14:26.9342346Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9342983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T03:14:26.9343580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T03:14:26.9344393Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:14:26.9345200Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:14:26.9345607Z ok (5.046s) 2022-11-23T03:14:26.9345761Z 2022-11-23T03:14:26.9346018Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9346368Z Ran 1 test in 5.046s 2022-11-23T03:14:26.9346527Z 2022-11-23T03:14:26.9346622Z OK 2022-11-23T03:14:26.9346754Z 2022-11-23T03:14:26.9346881Z Generating XML reports... 2022-11-23T03:14:26.9347448Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123025604.xml 2022-11-23T03:14:26.9348135Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9348598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9349253Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9349738Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9350211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf2vrmnp1 2022-11-23T03:14:26.9350759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf2vrmnp1/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9351377Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9351644Z 2022-11-23T03:14:26.9351757Z Running tests... 2022-11-23T03:14:26.9352203Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9352715Z test_sequence_num_incremented_gloo_default (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102282 2022-11-23T03:14:26.9353314Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102283 2022-11-23T03:14:26.9353974Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9354442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9355046Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9355524Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9355991Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq1no8_y3 2022-11-23T03:14:26.9356527Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq1no8_y3/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9357014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9357661Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9358122Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9358726Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9359208Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9359676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm1s9_9ua 2022-11-23T03:14:26.9360207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm1s9_9ua/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9360689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9361179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:14:26.9361674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:14:26.9362350Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9363062Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9363591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T03:14:26.9364085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T03:14:26.9364759Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:14:26.9365433Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:14:26.9365834Z ok (5.223s) 2022-11-23T03:14:26.9365982Z 2022-11-23T03:14:26.9366267Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9366679Z Ran 1 test in 5.223s 2022-11-23T03:14:26.9366840Z 2022-11-23T03:14:26.9366938Z OK 2022-11-23T03:14:26.9367079Z 2022-11-23T03:14:26.9367210Z Generating XML reports... 2022-11-23T03:14:26.9367877Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123025613.xml 2022-11-23T03:14:26.9368592Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9369118Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9369840Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9370419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9370986Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp588dal2c 2022-11-23T03:14:26.9371715Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp588dal2c/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9372473Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9372786Z 2022-11-23T03:14:26.9372889Z Running tests... 2022-11-23T03:14:26.9373416Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9374054Z test_sequence_num_incremented_gloo_subgroup (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102495 2022-11-23T03:14:26.9374715Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102496 2022-11-23T03:14:26.9375482Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9376036Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9376769Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9377350Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9377886Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp25cvj218 2022-11-23T03:14:26.9378519Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp25cvj218/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9379135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9379910Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9380459Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9381178Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9381757Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9382293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4tasqtor 2022-11-23T03:14:26.9382940Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4tasqtor/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9383557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9384040Z skip: Need at least 4 CUDA devices (4.715s) 2022-11-23T03:14:26.9384271Z 2022-11-23T03:14:26.9384614Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9385027Z Ran 1 test in 4.715s 2022-11-23T03:14:26.9385200Z 2022-11-23T03:14:26.9385307Z OK (skipped=1) 2022-11-23T03:14:26.9385438Z 2022-11-23T03:14:26.9385564Z Generating XML reports... 2022-11-23T03:14:26.9386135Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123025622.xml 2022-11-23T03:14:26.9386821Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9387366Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9387979Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9388464Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9388932Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnm3w_sf9 2022-11-23T03:14:26.9389442Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnm3w_sf9/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9390057Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9390318Z 2022-11-23T03:14:26.9390428Z Running tests... 2022-11-23T03:14:26.9390871Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9391400Z test_sequence_num_set_default_pg_gloo (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102696 2022-11-23T03:14:26.9426365Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102697 2022-11-23T03:14:26.9427155Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9427618Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9428220Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9428695Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9429154Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp4jo_9d4 2022-11-23T03:14:26.9429671Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp4jo_9d4/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9430155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9430783Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9431228Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9431821Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9432286Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9432734Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp25bbario 2022-11-23T03:14:26.9433256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp25bbario/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9433761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9434217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:14:26.9434701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:14:26.9435368Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9436067Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9436453Z ok (4.731s) 2022-11-23T03:14:26.9436598Z 2022-11-23T03:14:26.9436876Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9437196Z Ran 1 test in 4.731s 2022-11-23T03:14:26.9437334Z 2022-11-23T03:14:26.9437420Z OK 2022-11-23T03:14:26.9437552Z 2022-11-23T03:14:26.9437673Z Generating XML reports... 2022-11-23T03:14:26.9438241Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123025631.xml 2022-11-23T03:14:26.9438920Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9439477Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9440075Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9440548Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9440986Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvrd468n0 2022-11-23T03:14:26.9441509Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvrd468n0/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9442106Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9442366Z 2022-11-23T03:14:26.9442471Z Running tests... 2022-11-23T03:14:26.9442895Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9443409Z test_sequence_num_set_gloo_new_group (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102903 2022-11-23T03:14:26.9443993Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102904 2022-11-23T03:14:26.9444614Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9445067Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9445664Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9446133Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9446589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1a5cmd_z 2022-11-23T03:14:26.9447116Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1a5cmd_z/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9447623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9448331Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9448767Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9449480Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9450045Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9450600Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq_immfoe 2022-11-23T03:14:26.9452482Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq_immfoe/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9453083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9453656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:14:26.9454441Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9455076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:14:26.9455850Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9456472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T03:14:26.9457037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T03:14:26.9457819Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:14:26.9458614Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:14:26.9459066Z ok (4.716s) 2022-11-23T03:14:26.9459226Z 2022-11-23T03:14:26.9459633Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9460005Z Ran 1 test in 4.716s 2022-11-23T03:14:26.9460179Z 2022-11-23T03:14:26.9460277Z OK 2022-11-23T03:14:26.9460426Z 2022-11-23T03:14:26.9460571Z Generating XML reports... 2022-11-23T03:14:26.9461226Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123025640.xml 2022-11-23T03:14:26.9462034Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9462574Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9463269Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9463824Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9464368Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnmvqqhao 2022-11-23T03:14:26.9465090Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnmvqqhao/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9465738Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9465998Z 2022-11-23T03:14:26.9466099Z Running tests... 2022-11-23T03:14:26.9466517Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9466988Z test_tensor_dtype_complex (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103116 2022-11-23T03:14:26.9467499Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103117 2022-11-23T03:14:26.9468121Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9468571Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9469162Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9469632Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9470095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy6iup65v 2022-11-23T03:14:26.9470611Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy6iup65v/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9471084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9471710Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9472150Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9472739Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9473204Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9473660Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm_zr22j2 2022-11-23T03:14:26.9474178Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm_zr22j2/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9474655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9475135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:14:26.9475615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:14:26.9476276Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9476964Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9477352Z ok (4.622s) 2022-11-23T03:14:26.9477495Z 2022-11-23T03:14:26.9477849Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9478162Z Ran 1 test in 4.622s 2022-11-23T03:14:26.9478308Z 2022-11-23T03:14:26.9478391Z OK 2022-11-23T03:14:26.9478508Z 2022-11-23T03:14:26.9478621Z Generating XML reports... 2022-11-23T03:14:26.9479156Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123025649.xml 2022-11-23T03:14:26.9479810Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9480237Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9480817Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9481277Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9481729Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_3c5h4yy 2022-11-23T03:14:26.9482318Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_3c5h4yy/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9482931Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9483180Z 2022-11-23T03:14:26.9483276Z Running tests... 2022-11-23T03:14:26.9483677Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9484158Z test_tensor_dtype_mismatch (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103323 2022-11-23T03:14:26.9484654Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103324 2022-11-23T03:14:26.9485248Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9485679Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9486267Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9486717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9487159Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc827thek 2022-11-23T03:14:26.9487672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc827thek/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9488218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9488825Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9489264Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9489843Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9490291Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9490740Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptpb5toy1 2022-11-23T03:14:26.9491250Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptpb5toy1/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9491739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9492195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:14:26.9492843Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9493348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:14:26.9493986Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9495039Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T03:14:26.9495711Z warnings.warn( 2022-11-23T03:14:26.9496601Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T03:14:26.9497200Z warnings.warn( 2022-11-23T03:14:26.9498081Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T03:14:26.9498674Z warnings.warn( 2022-11-23T03:14:26.9499592Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T03:14:26.9500196Z warnings.warn( 2022-11-23T03:14:26.9500428Z ok (4.627s) 2022-11-23T03:14:26.9500564Z 2022-11-23T03:14:26.9500838Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9501154Z Ran 1 test in 4.627s 2022-11-23T03:14:26.9501303Z 2022-11-23T03:14:26.9501387Z OK 2022-11-23T03:14:26.9501511Z 2022-11-23T03:14:26.9501627Z Generating XML reports... 2022-11-23T03:14:26.9502154Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221123025657.xml 2022-11-23T03:14:26.9502809Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9503249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9503833Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9504290Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9504732Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqeewthao 2022-11-23T03:14:26.9505247Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqeewthao/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9505835Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9506071Z 2022-11-23T03:14:26.9506175Z Running tests... 2022-11-23T03:14:26.9506586Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9507080Z test_allgather_work_wait_cpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103530 2022-11-23T03:14:26.9507595Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103531 2022-11-23T03:14:26.9508208Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9508647Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9509221Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9509664Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9510106Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ikqg69j 2022-11-23T03:14:26.9510614Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ikqg69j/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9511100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9511719Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9512216Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9512796Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9513240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9513686Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0afz5w7z 2022-11-23T03:14:26.9514195Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0afz5w7z/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9514683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9515151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:14:26.9515624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:14:26.9516331Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9517015Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9517936Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9518655Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9519519Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9520234Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9520551Z ok (4.728s) 2022-11-23T03:14:26.9520692Z 2022-11-23T03:14:26.9520960Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9521276Z Ran 1 test in 4.729s 2022-11-23T03:14:26.9521427Z 2022-11-23T03:14:26.9521514Z OK 2022-11-23T03:14:26.9521638Z 2022-11-23T03:14:26.9521740Z Generating XML reports... 2022-11-23T03:14:26.9522294Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123025706.xml 2022-11-23T03:14:26.9522957Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9523394Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9523973Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9524425Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9524878Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy9xr01eg 2022-11-23T03:14:26.9525375Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy9xr01eg/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9525959Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9526213Z 2022-11-23T03:14:26.9526317Z Running tests... 2022-11-23T03:14:26.9526730Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9527223Z test_allgather_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103737 2022-11-23T03:14:26.9527865Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103738 2022-11-23T03:14:26.9528480Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9528995Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9529565Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9530018Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9530465Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw7ce4xbx 2022-11-23T03:14:26.9530972Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw7ce4xbx/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9531458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9532071Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9532503Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9533127Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9533593Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9534031Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphqoxiqs3 2022-11-23T03:14:26.9534536Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphqoxiqs3/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9535020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9535483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:14:26.9535954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:14:26.9536599Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9537269Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9538198Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9538911Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9539771Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9540476Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9540792Z ok (5.191s) 2022-11-23T03:14:26.9540928Z 2022-11-23T03:14:26.9541193Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9541518Z Ran 1 test in 5.192s 2022-11-23T03:14:26.9541669Z 2022-11-23T03:14:26.9541741Z OK 2022-11-23T03:14:26.9541864Z 2022-11-23T03:14:26.9541978Z Generating XML reports... 2022-11-23T03:14:26.9542525Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123025715.xml 2022-11-23T03:14:26.9543198Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9543627Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9544202Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9544651Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9545086Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2blc39d5 2022-11-23T03:14:26.9545595Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2blc39d5/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9546247Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9546492Z 2022-11-23T03:14:26.9546590Z Running tests... 2022-11-23T03:14:26.9546996Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9547487Z test_allreduce_work_wait_cpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103948 2022-11-23T03:14:26.9548004Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103949 2022-11-23T03:14:26.9548612Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9549031Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9549612Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9550130Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9550572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb_c_chp8 2022-11-23T03:14:26.9551082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb_c_chp8/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9551728Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9552161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9552719Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9553169Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9553610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps_3r7lky 2022-11-23T03:14:26.9554122Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps_3r7lky/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9554603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9555055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9555517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:14:26.9555973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:14:26.9556620Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9557301Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9558234Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9558950Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9559816Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9560531Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9561400Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9562109Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9563037Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9563747Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9564049Z ok (4.615s) 2022-11-23T03:14:26.9564186Z 2022-11-23T03:14:26.9564456Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9564777Z Ran 1 test in 4.615s 2022-11-23T03:14:26.9564927Z 2022-11-23T03:14:26.9565013Z OK 2022-11-23T03:14:26.9565136Z 2022-11-23T03:14:26.9565252Z Generating XML reports... 2022-11-23T03:14:26.9565803Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123025724.xml 2022-11-23T03:14:26.9566521Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9566953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9567536Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9568053Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9568490Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp638d4h8q 2022-11-23T03:14:26.9568997Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp638d4h8q/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9569588Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9569835Z 2022-11-23T03:14:26.9569934Z Running tests... 2022-11-23T03:14:26.9570326Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9570825Z test_allreduce_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104155 2022-11-23T03:14:26.9571340Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104156 2022-11-23T03:14:26.9571955Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9572393Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9572969Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9573425Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9573864Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ejn3cvg 2022-11-23T03:14:26.9574360Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ejn3cvg/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9574857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9575479Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9575906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9576479Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9576931Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9577376Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppx431ihy 2022-11-23T03:14:26.9577875Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppx431ihy/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9578359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9578823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:14:26.9579543Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9580049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:14:26.9580692Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9581619Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9582326Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9583234Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9583943Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9584802Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9585508Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9586365Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9587069Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9587385Z ok (5.117s) 2022-11-23T03:14:26.9587528Z 2022-11-23T03:14:26.9587802Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9588122Z Ran 1 test in 5.118s 2022-11-23T03:14:26.9588272Z 2022-11-23T03:14:26.9588355Z OK 2022-11-23T03:14:26.9588467Z 2022-11-23T03:14:26.9588582Z Generating XML reports... 2022-11-23T03:14:26.9589133Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123025733.xml 2022-11-23T03:14:26.9589795Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9590226Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9590800Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9591255Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9591704Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpub38jamn 2022-11-23T03:14:26.9592206Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpub38jamn/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9592792Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9593039Z 2022-11-23T03:14:26.9593140Z Running tests... 2022-11-23T03:14:26.9593546Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9594041Z test_broadcast_work_wait_cpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104366 2022-11-23T03:14:26.9594555Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104367 2022-11-23T03:14:26.9595164Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9595590Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9596249Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9596707Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9597150Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu5zd2qbz 2022-11-23T03:14:26.9597663Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu5zd2qbz/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9598309Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9598742Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9599314Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9599748Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9600245Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxnk1ninm 2022-11-23T03:14:26.9600773Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxnk1ninm/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9601257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9601714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9602177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:14:26.9602822Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9603315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:14:26.9603949Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9604875Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9605585Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9606438Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9607140Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9607458Z ok (4.775s) 2022-11-23T03:14:26.9607593Z 2022-11-23T03:14:26.9607912Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9608222Z Ran 1 test in 4.776s 2022-11-23T03:14:26.9608375Z 2022-11-23T03:14:26.9608452Z OK 2022-11-23T03:14:26.9608578Z 2022-11-23T03:14:26.9608690Z Generating XML reports... 2022-11-23T03:14:26.9609239Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123025742.xml 2022-11-23T03:14:26.9609910Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9610341Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9610914Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9611368Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9611797Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprpgu192d 2022-11-23T03:14:26.9612306Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprpgu192d/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9612898Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9613221Z 2022-11-23T03:14:26.9613318Z Running tests... 2022-11-23T03:14:26.9613725Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9614217Z test_broadcast_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104573 2022-11-23T03:14:26.9614721Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104574 2022-11-23T03:14:26.9615315Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9615744Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9616310Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9616755Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9617249Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb20sl5xg 2022-11-23T03:14:26.9617756Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb20sl5xg/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9618237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9618846Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9619266Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9619835Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9620284Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9620718Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpklj28twa 2022-11-23T03:14:26.9621227Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpklj28twa/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9621711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9622168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:14:26.9622800Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9623304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:14:26.9623928Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9624844Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9625553Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9626403Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9627107Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9627417Z ok (5.211s) 2022-11-23T03:14:26.9627551Z 2022-11-23T03:14:26.9627814Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9628114Z Ran 1 test in 5.211s 2022-11-23T03:14:26.9628260Z 2022-11-23T03:14:26.9628343Z OK 2022-11-23T03:14:26.9628463Z 2022-11-23T03:14:26.9628577Z Generating XML reports... 2022-11-23T03:14:26.9629129Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123025751.xml 2022-11-23T03:14:26.9629855Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9630290Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9630868Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9631311Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9631750Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdcoch1_0 2022-11-23T03:14:26.9632265Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdcoch1_0/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9632843Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9633090Z 2022-11-23T03:14:26.9633191Z Running tests... 2022-11-23T03:14:26.9633596Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9634157Z test_consecutive_comm_work_wait_cpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104784 2022-11-23T03:14:26.9634668Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104785 2022-11-23T03:14:26.9635282Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9635716Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9636295Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9636750Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9637193Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcpl4ndg0 2022-11-23T03:14:26.9637703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcpl4ndg0/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9638196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9638801Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9639235Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9639812Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9640260Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9640703Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt5936pfh 2022-11-23T03:14:26.9641209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt5936pfh/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9641693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9642151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:14:26.9642626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:14:26.9643273Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9643946Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9644872Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9645583Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9646444Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9647223Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9648196Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9648905Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9649753Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9650523Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9651391Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9652094Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9652955Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9653662Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9654526Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9655234Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9656086Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9656795Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9657111Z ok (4.826s) 2022-11-23T03:14:26.9657247Z 2022-11-23T03:14:26.9657504Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9657817Z Ran 1 test in 4.826s 2022-11-23T03:14:26.9657970Z 2022-11-23T03:14:26.9658054Z OK 2022-11-23T03:14:26.9658177Z 2022-11-23T03:14:26.9658290Z Generating XML reports... 2022-11-23T03:14:26.9658839Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123025800.xml 2022-11-23T03:14:26.9659503Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9659940Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9660507Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9660967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9661410Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfepxnqjh 2022-11-23T03:14:26.9661922Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfepxnqjh/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9662509Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9662755Z 2022-11-23T03:14:26.9662856Z Running tests... 2022-11-23T03:14:26.9663335Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9663828Z test_consecutive_comm_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104991 2022-11-23T03:14:26.9664344Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104992 2022-11-23T03:14:26.9664954Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9665388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9665966Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9666416Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9666854Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj7yx9lle 2022-11-23T03:14:26.9667418Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj7yx9lle/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9667898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9668522Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9668957Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9669530Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9669980Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9670427Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ij9yfei 2022-11-23T03:14:26.9670936Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ij9yfei/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9671410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9671876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:14:26.9672522Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9673034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:14:26.9673673Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9674600Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9675308Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9676171Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9676875Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9677729Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9678424Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9679282Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9680055Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9680916Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9681619Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9682472Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9683182Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9684088Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9684806Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9685666Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9686375Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9686693Z ok (5.416s) 2022-11-23T03:14:26.9686820Z 2022-11-23T03:14:26.9687088Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9687409Z Ran 1 test in 5.417s 2022-11-23T03:14:26.9687565Z 2022-11-23T03:14:26.9687648Z OK 2022-11-23T03:14:26.9687832Z 2022-11-23T03:14:26.9687950Z Generating XML reports... 2022-11-23T03:14:26.9688504Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123025809.xml 2022-11-23T03:14:26.9689175Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9689598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9690178Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9690636Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9691082Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprzfpm3s9 2022-11-23T03:14:26.9691599Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprzfpm3s9/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9692189Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9692438Z 2022-11-23T03:14:26.9692539Z Running tests... 2022-11-23T03:14:26.9692932Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9693438Z test_nested_comm_tensor_wrapping (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105202 2022-11-23T03:14:26.9693955Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105203 2022-11-23T03:14:26.9694562Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9694995Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9695576Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9696033Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9696481Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpke5byebm 2022-11-23T03:14:26.9697053Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpke5byebm/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9697704Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9698137Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9698714Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9699179Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9699622Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuzyusz_u 2022-11-23T03:14:26.9700140Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuzyusz_u/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9700671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9701139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9701605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:14:26.9702077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:14:26.9702725Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9703403Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9704331Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9705045Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9705907Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9706621Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9707469Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9708173Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9709035Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9709744Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9710059Z ok (5.532s) 2022-11-23T03:14:26.9710203Z 2022-11-23T03:14:26.9710471Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9710786Z Ran 1 test in 5.532s 2022-11-23T03:14:26.9710934Z 2022-11-23T03:14:26.9711018Z OK 2022-11-23T03:14:26.9711144Z 2022-11-23T03:14:26.9711248Z Generating XML reports... 2022-11-23T03:14:26.9711799Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123025818.xml 2022-11-23T03:14:26.9712472Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9712907Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9713566Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9714021Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9714464Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxa_0firp 2022-11-23T03:14:26.9714959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxa_0firp/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9715542Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9715794Z 2022-11-23T03:14:26.9715895Z Running tests... 2022-11-23T03:14:26.9716302Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9716794Z test_scatter_work_wait_cpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105409 2022-11-23T03:14:26.9717358Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105410 2022-11-23T03:14:26.9718171Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9718701Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9719386Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9719930Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9720466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbqhekhc8 2022-11-23T03:14:26.9721076Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbqhekhc8/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9721665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9722409Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9722937Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9723619Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9724163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9724694Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ocefqhu 2022-11-23T03:14:26.9725305Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ocefqhu/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9725889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9726451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:14:26.9727016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:14:26.9727974Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9728811Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9729928Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9730780Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9731817Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9732660Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9733139Z ok (5.429s) 2022-11-23T03:14:26.9733305Z 2022-11-23T03:14:26.9733628Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9734008Z Ran 1 test in 5.429s 2022-11-23T03:14:26.9734191Z 2022-11-23T03:14:26.9734280Z OK 2022-11-23T03:14:26.9734428Z 2022-11-23T03:14:26.9734568Z Generating XML reports... 2022-11-23T03:14:26.9735222Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123025828.xml 2022-11-23T03:14:26.9735949Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9736389Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9736978Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9737439Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9737931Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgg9jg3ey 2022-11-23T03:14:26.9738452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgg9jg3ey/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9739043Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9739297Z 2022-11-23T03:14:26.9739398Z Running tests... 2022-11-23T03:14:26.9739804Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9740297Z test_scatter_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105616 2022-11-23T03:14:26.9740802Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105617 2022-11-23T03:14:26.9741413Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9741840Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9742420Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9742875Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9743322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj3wooup4 2022-11-23T03:14:26.9743831Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj3wooup4/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9744483Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9744916Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9745483Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9745943Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9746404Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplvb9bf_x 2022-11-23T03:14:26.9746920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplvb9bf_x/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9747408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9747864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9748330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:14:26.9748965Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9749476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:14:26.9750117Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:26.9751113Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9751827Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9752690Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:14:26.9753405Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:14:26.9753726Z ok (5.381s) 2022-11-23T03:14:26.9753868Z 2022-11-23T03:14:26.9754134Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9754440Z Ran 1 test in 5.382s 2022-11-23T03:14:26.9754647Z 2022-11-23T03:14:26.9754733Z OK 2022-11-23T03:14:26.9754859Z 2022-11-23T03:14:26.9754976Z Generating XML reports... 2022-11-23T03:14:26.9755532Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221123025838.xml 2022-11-23T03:14:26.9756191Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9756621Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9757202Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9757646Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9758095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuj30ew41 2022-11-23T03:14:26.9758608Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuj30ew41/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9759201Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9759453Z 2022-11-23T03:14:26.9759553Z Running tests... 2022-11-23T03:14:26.9759957Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9760375Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9761074Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105827 2022-11-23T03:14:26.9761613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105828 2022-11-23T03:14:26.9762224Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9762661Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9763243Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9763701Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9764144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp65p2loet 2022-11-23T03:14:26.9764658Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp65p2loet/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9765131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9765749Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9766222Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9766812Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9767267Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9767863Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj992xgmq 2022-11-23T03:14:26.9768376Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj992xgmq/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9768849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9769182Z ok (7.324s) 2022-11-23T03:14:26.9769189Z 2022-11-23T03:14:26.9769471Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9769573Z Ran 1 test in 7.324s 2022-11-23T03:14:26.9769579Z 2022-11-23T03:14:26.9769664Z OK 2022-11-23T03:14:26.9769670Z 2022-11-23T03:14:26.9769785Z Generating XML reports... 2022-11-23T03:14:26.9770245Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123025847.xml 2022-11-23T03:14:26.9770679Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9770848Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9771234Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9771414Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9771651Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6r82iu05 2022-11-23T03:14:26.9771899Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6r82iu05/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9772212Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9772219Z 2022-11-23T03:14:26.9772320Z Running tests... 2022-11-23T03:14:26.9772590Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9772811Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9773125Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106044 2022-11-23T03:14:26.9773331Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106045 2022-11-23T03:14:26.9773692Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9773859Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9774238Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9774414Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9774648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp783j9vh_ 2022-11-23T03:14:26.9774891Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp783j9vh_/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9775268Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9775432Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9775815Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9775998Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9776233Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptq7vvoih 2022-11-23T03:14:26.9776480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptq7vvoih/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9776693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9776908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9777061Z ok (7.421s) 2022-11-23T03:14:26.9777071Z 2022-11-23T03:14:26.9777342Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9777447Z Ran 1 test in 7.422s 2022-11-23T03:14:26.9777453Z 2022-11-23T03:14:26.9777538Z OK 2022-11-23T03:14:26.9777543Z 2022-11-23T03:14:26.9777659Z Generating XML reports... 2022-11-23T03:14:26.9778104Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123025858.xml 2022-11-23T03:14:26.9778476Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9778642Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9779010Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9779189Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9779468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7f8hujbt 2022-11-23T03:14:26.9779723Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7f8hujbt/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9780041Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9780047Z 2022-11-23T03:14:26.9780146Z Running tests... 2022-11-23T03:14:26.9780411Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9780633Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9780930Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106261 2022-11-23T03:14:26.9781137Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106262 2022-11-23T03:14:26.9781509Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9781673Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9782056Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9782236Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9782470Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvymes_dg 2022-11-23T03:14:26.9782716Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvymes_dg/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9782927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9783304Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9783469Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9783855Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9784036Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9784259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe98r5a60 2022-11-23T03:14:26.9784505Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe98r5a60/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9784718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9784936Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9785152Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9785370Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9785588Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9786593Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T03:14:26.9786700Z warnings.warn( 2022-11-23T03:14:26.9786916Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9787851Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T03:14:26.9787997Z warnings.warn( 2022-11-23T03:14:26.9788222Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9788436Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9788652Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9788872Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9789091Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9789305Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9789518Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9789608Z ok (7.430s) 2022-11-23T03:14:26.9789614Z 2022-11-23T03:14:26.9789890Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9789999Z Ran 1 test in 7.430s 2022-11-23T03:14:26.9790005Z 2022-11-23T03:14:26.9790091Z OK 2022-11-23T03:14:26.9790097Z 2022-11-23T03:14:26.9790214Z Generating XML reports... 2022-11-23T03:14:26.9790666Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123025910.xml 2022-11-23T03:14:26.9791040Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9791193Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9791575Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9791752Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9791989Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiy3lcu9u 2022-11-23T03:14:26.9792239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiy3lcu9u/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9792556Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9792562Z 2022-11-23T03:14:26.9792662Z Running tests... 2022-11-23T03:14:26.9792930Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9793151Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9793442Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106478 2022-11-23T03:14:26.9793648Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106479 2022-11-23T03:14:26.9794021Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9794185Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9794630Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9794809Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9795048Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzcch32h8 2022-11-23T03:14:26.9795295Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzcch32h8/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9795507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9795876Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9796039Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9796421Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9796650Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9796875Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnuhle2ea 2022-11-23T03:14:26.9797130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnuhle2ea/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9797344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9797563Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9797781Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9797995Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9798212Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9799150Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T03:14:26.9799258Z warnings.warn( 2022-11-23T03:14:26.9799476Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9800410Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T03:14:26.9800515Z warnings.warn( 2022-11-23T03:14:26.9800735Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9800954Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9801170Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9801386Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9801603Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9801820Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9802039Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9802130Z ok (7.328s) 2022-11-23T03:14:26.9802136Z 2022-11-23T03:14:26.9802405Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9802508Z Ran 1 test in 7.328s 2022-11-23T03:14:26.9802514Z 2022-11-23T03:14:26.9802599Z OK 2022-11-23T03:14:26.9802655Z 2022-11-23T03:14:26.9802778Z Generating XML reports... 2022-11-23T03:14:26.9803219Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123025921.xml 2022-11-23T03:14:26.9803591Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9803755Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9804139Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9804315Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9804553Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr1um34n4 2022-11-23T03:14:26.9804800Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr1um34n4/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9805152Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9805163Z 2022-11-23T03:14:26.9805267Z Running tests... 2022-11-23T03:14:26.9805534Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9805779Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9806230Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106695 2022-11-23T03:14:26.9806436Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106696 2022-11-23T03:14:26.9806805Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9806970Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9807352Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9807536Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9807836Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphl2qud5b 2022-11-23T03:14:26.9808088Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphl2qud5b/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9808303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9808681Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9808844Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9809225Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9809390Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9809628Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmcuz0v1i 2022-11-23T03:14:26.9809880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmcuz0v1i/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9810094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9810315Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9810537Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9810754Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9810972Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9811060Z ok (7.420s) 2022-11-23T03:14:26.9811066Z 2022-11-23T03:14:26.9811333Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9811437Z Ran 1 test in 7.420s 2022-11-23T03:14:26.9811522Z 2022-11-23T03:14:26.9811613Z OK 2022-11-23T03:14:26.9811619Z 2022-11-23T03:14:26.9811732Z Generating XML reports... 2022-11-23T03:14:26.9812186Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123025933.xml 2022-11-23T03:14:26.9812556Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9812722Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9813105Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9813284Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9813516Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7k7wsz1q 2022-11-23T03:14:26.9813761Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7k7wsz1q/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9814125Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9814132Z 2022-11-23T03:14:26.9814220Z Running tests... 2022-11-23T03:14:26.9814492Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9814735Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9815183Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106912 2022-11-23T03:14:26.9815394Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106913 2022-11-23T03:14:26.9815766Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9815935Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9816324Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9816502Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9816736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyxsqvo78 2022-11-23T03:14:26.9816984Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyxsqvo78/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9817356Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9817521Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9817901Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9818080Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9818316Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkve885sk 2022-11-23T03:14:26.9818568Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkve885sk/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9818784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9819002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9819222Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9819440Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9819654Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9819858Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9819947Z ok (7.236s) 2022-11-23T03:14:26.9819953Z 2022-11-23T03:14:26.9820226Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9820384Z Ran 1 test in 7.236s 2022-11-23T03:14:26.9820391Z 2022-11-23T03:14:26.9820478Z OK 2022-11-23T03:14:26.9820484Z 2022-11-23T03:14:26.9820601Z Generating XML reports... 2022-11-23T03:14:26.9821052Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123025944.xml 2022-11-23T03:14:26.9821422Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9821587Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9821962Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9822137Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9822375Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq7fofl_9 2022-11-23T03:14:26.9822667Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq7fofl_9/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9822987Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9822993Z 2022-11-23T03:14:26.9823094Z Running tests... 2022-11-23T03:14:26.9823359Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9823582Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9824056Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107129 2022-11-23T03:14:26.9824263Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107130 2022-11-23T03:14:26.9824635Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9824801Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9825175Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9825353Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9825587Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl8j_faxk 2022-11-23T03:14:26.9825831Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl8j_faxk/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9826045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9826416Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9826582Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9826965Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9827148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9827380Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ce3jmtk 2022-11-23T03:14:26.9827629Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ce3jmtk/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9827839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9828054Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9828273Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9829028Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T03:14:26.9829838Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T03:14:26.9830062Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9830277Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9830374Z ok (7.321s) 2022-11-23T03:14:26.9830415Z 2022-11-23T03:14:26.9830689Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9830791Z Ran 1 test in 7.321s 2022-11-23T03:14:26.9830797Z 2022-11-23T03:14:26.9830883Z OK 2022-11-23T03:14:26.9830889Z 2022-11-23T03:14:26.9831001Z Generating XML reports... 2022-11-23T03:14:26.9831451Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123025956.xml 2022-11-23T03:14:26.9831825Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9831988Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9832368Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9832546Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9832787Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp391cv1nv 2022-11-23T03:14:26.9833033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp391cv1nv/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9833333Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9833351Z 2022-11-23T03:14:26.9833438Z Running tests... 2022-11-23T03:14:26.9833702Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9833926Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9834400Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107346 2022-11-23T03:14:26.9834613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107347 2022-11-23T03:14:26.9834986Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9835158Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9835539Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9835714Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9835950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqjuatf8h 2022-11-23T03:14:26.9836200Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqjuatf8h/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9836412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9836787Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9836953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9837396Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9837573Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9837809Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjrq473vr 2022-11-23T03:14:26.9838060Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjrq473vr/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9838275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9838368Z ok (7.115s) 2022-11-23T03:14:26.9838374Z 2022-11-23T03:14:26.9838639Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9838727Z Ran 1 test in 7.116s 2022-11-23T03:14:26.9838749Z 2022-11-23T03:14:26.9838821Z OK 2022-11-23T03:14:26.9838826Z 2022-11-23T03:14:26.9838943Z Generating XML reports... 2022-11-23T03:14:26.9839438Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123030008.xml 2022-11-23T03:14:26.9839819Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9839983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9840364Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9840542Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9840775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4jb41g25 2022-11-23T03:14:26.9841018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4jb41g25/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9841335Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9841341Z 2022-11-23T03:14:26.9841443Z Running tests... 2022-11-23T03:14:26.9841713Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9841931Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9842247Z Checkpointing should work with static graph in the case of checkpointing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107563 2022-11-23T03:14:26.9842455Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107564 2022-11-23T03:14:26.9842825Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9842991Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9843376Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9843555Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9843793Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcr8yjc57 2022-11-23T03:14:26.9844039Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcr8yjc57/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9844398Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9844564Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9844946Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9845124Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9845355Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6xgrpo1i 2022-11-23T03:14:26.9845602Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6xgrpo1i/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9845817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9846086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9846307Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9846526Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9846739Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9846959Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9847048Z ok (8.177s) 2022-11-23T03:14:26.9847055Z 2022-11-23T03:14:26.9847326Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9847426Z Ran 1 test in 8.177s 2022-11-23T03:14:26.9847432Z 2022-11-23T03:14:26.9847520Z OK 2022-11-23T03:14:26.9847525Z 2022-11-23T03:14:26.9847639Z Generating XML reports... 2022-11-23T03:14:26.9848292Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123030019.xml 2022-11-23T03:14:26.9848671Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9848834Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9849217Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9849382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9849616Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzhsfe33d 2022-11-23T03:14:26.9849862Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzhsfe33d/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9850175Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9850186Z 2022-11-23T03:14:26.9850289Z Running tests... 2022-11-23T03:14:26.9850561Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9850797Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9851110Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107780 2022-11-23T03:14:26.9851317Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107781 2022-11-23T03:14:26.9851688Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9851851Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9852235Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9852417Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9852652Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwlym5rrf 2022-11-23T03:14:26.9852901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwlym5rrf/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9853115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9853486Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9853652Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9854034Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9854212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9854444Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4prkl7e3 2022-11-23T03:14:26.9854773Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4prkl7e3/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9854973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9855734Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T03:14:26.9856517Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T03:14:26.9857471Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T03:14:26.9857574Z warnings.warn( 2022-11-23T03:14:26.9857794Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9858731Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T03:14:26.9858837Z warnings.warn( 2022-11-23T03:14:26.9859058Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9859272Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9859491Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9859583Z ok (7.325s) 2022-11-23T03:14:26.9859589Z 2022-11-23T03:14:26.9859857Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9859945Z Ran 1 test in 7.325s 2022-11-23T03:14:26.9859963Z 2022-11-23T03:14:26.9860034Z OK 2022-11-23T03:14:26.9860043Z 2022-11-23T03:14:26.9860161Z Generating XML reports... 2022-11-23T03:14:26.9860610Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123030032.xml 2022-11-23T03:14:26.9860983Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9861149Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9861527Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9861706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9861939Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpie_v0he5 2022-11-23T03:14:26.9862185Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpie_v0he5/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9862498Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9862558Z 2022-11-23T03:14:26.9862662Z Running tests... 2022-11-23T03:14:26.9862932Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9863168Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9863478Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107997 2022-11-23T03:14:26.9863686Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107998 2022-11-23T03:14:26.9864057Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9864220Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9864602Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9864828Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9865070Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2sf3ryg_ 2022-11-23T03:14:26.9865318Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2sf3ryg_/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9865696Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9865848Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9866230Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9866408Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9866648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwe3anpii 2022-11-23T03:14:26.9866901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwe3anpii/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9867118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9867336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9868265Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T03:14:26.9868366Z warnings.warn( 2022-11-23T03:14:26.9868584Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9869519Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T03:14:26.9869622Z warnings.warn( 2022-11-23T03:14:26.9869842Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9870059Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9870276Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9870365Z ok (7.234s) 2022-11-23T03:14:26.9870371Z 2022-11-23T03:14:26.9870639Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9870739Z Ran 1 test in 7.234s 2022-11-23T03:14:26.9870745Z 2022-11-23T03:14:26.9870830Z OK 2022-11-23T03:14:26.9870836Z 2022-11-23T03:14:26.9871006Z Generating XML reports... 2022-11-23T03:14:26.9871460Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123030043.xml 2022-11-23T03:14:26.9871831Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9871995Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9872375Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9872553Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9872775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ddl0jix 2022-11-23T03:14:26.9873025Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ddl0jix/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9873399Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9873410Z 2022-11-23T03:14:26.9873516Z Running tests... 2022-11-23T03:14:26.9873784Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9874022Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9874308Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108214 2022-11-23T03:14:26.9874545Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108215 2022-11-23T03:14:26.9874920Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9875083Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9875465Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9875650Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9875890Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv0_nyf53 2022-11-23T03:14:26.9876137Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv0_nyf53/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9876350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9876745Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9876914Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9877298Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9877477Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9877715Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbq5hy5bg 2022-11-23T03:14:26.9877968Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbq5hy5bg/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9878185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9878388Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9878606Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9878837Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9879056Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9879149Z ok (7.221s) 2022-11-23T03:14:26.9879155Z 2022-11-23T03:14:26.9879426Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9879529Z Ran 1 test in 7.221s 2022-11-23T03:14:26.9879535Z 2022-11-23T03:14:26.9879674Z OK 2022-11-23T03:14:26.9879683Z 2022-11-23T03:14:26.9879804Z Generating XML reports... 2022-11-23T03:14:26.9880257Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123030054.xml 2022-11-23T03:14:26.9880627Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9880792Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9881177Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9881356Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9881589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp81hkdckw 2022-11-23T03:14:26.9881833Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp81hkdckw/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9882195Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9882203Z 2022-11-23T03:14:26.9882305Z Running tests... 2022-11-23T03:14:26.9882573Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9882806Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9883089Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108431 2022-11-23T03:14:26.9883283Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108432 2022-11-23T03:14:26.9883654Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9883819Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9884199Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9884387Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9884622Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf7cqr6vx 2022-11-23T03:14:26.9884869Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf7cqr6vx/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9885242Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9885407Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9885790Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9885969Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9886208Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5p6f72dw 2022-11-23T03:14:26.9886463Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5p6f72dw/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9886686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9886907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9887127Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9887352Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9887571Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9887860Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9888076Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9888298Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9888588Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9888790Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9888883Z ok (8.128s) 2022-11-23T03:14:26.9888889Z 2022-11-23T03:14:26.9889164Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9889266Z Ran 1 test in 8.128s 2022-11-23T03:14:26.9889272Z 2022-11-23T03:14:26.9889360Z OK 2022-11-23T03:14:26.9889365Z 2022-11-23T03:14:26.9889481Z Generating XML reports... 2022-11-23T03:14:26.9889933Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123030105.xml 2022-11-23T03:14:26.9890307Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9890474Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9890913Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9891096Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9891334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptr4orp39 2022-11-23T03:14:26.9891588Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptr4orp39/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9891906Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9891913Z 2022-11-23T03:14:26.9892016Z Running tests... 2022-11-23T03:14:26.9892283Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9892486Z test_ddp_comm_hook_future_passing_cpu (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9892796Z This unit test verifies whether the Future object is passed properly. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108648 2022-11-23T03:14:26.9893007Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108649 2022-11-23T03:14:26.9893381Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9893547Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9893915Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9894092Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9894329Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzb6vnqfm 2022-11-23T03:14:26.9894582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzb6vnqfm/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9894798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9895177Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9895345Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9895731Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9895914Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9896150Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5dm2siiq 2022-11-23T03:14:26.9896398Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5dm2siiq/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9896613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9896703Z ok (5.536s) 2022-11-23T03:14:26.9896709Z 2022-11-23T03:14:26.9896979Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9897138Z Ran 1 test in 5.536s 2022-11-23T03:14:26.9897145Z 2022-11-23T03:14:26.9897231Z OK 2022-11-23T03:14:26.9897236Z 2022-11-23T03:14:26.9897355Z Generating XML reports... 2022-11-23T03:14:26.9897807Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123030118.xml 2022-11-23T03:14:26.9898181Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9898347Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9898730Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9898897Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9899131Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwi0op0ov 2022-11-23T03:14:26.9899420Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwi0op0ov/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9899743Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9899749Z 2022-11-23T03:14:26.9899849Z Running tests... 2022-11-23T03:14:26.9900115Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9900328Z test_ddp_comm_hook_future_passing_gpu_gloo (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9900659Z This unit test verifies whether the Future object is passed properly using gloo backend. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108859 2022-11-23T03:14:26.9900864Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108860 2022-11-23T03:14:26.9901236Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9901405Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9901794Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9901973Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9902207Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0syqhf8e 2022-11-23T03:14:26.9902453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0syqhf8e/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9902824Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9902990Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9903373Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9903552Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9903790Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj315oira 2022-11-23T03:14:26.9904038Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj315oira/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9904254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9904455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9904548Z ok (5.999s) 2022-11-23T03:14:26.9904555Z 2022-11-23T03:14:26.9904824Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9904928Z Ran 1 test in 6.000s 2022-11-23T03:14:26.9904934Z 2022-11-23T03:14:26.9905019Z OK 2022-11-23T03:14:26.9905024Z 2022-11-23T03:14:26.9905139Z Generating XML reports... 2022-11-23T03:14:26.9905586Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123030128.xml 2022-11-23T03:14:26.9905959Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9906177Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9906561Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9906742Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9906978Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1e4pry8w 2022-11-23T03:14:26.9907225Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1e4pry8w/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9907538Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9907544Z 2022-11-23T03:14:26.9907647Z Running tests... 2022-11-23T03:14:26.9907913Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9908156Z test_ddp_comm_hook_register_just_once (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9908485Z DDP communication hook can only be registered once. This test validates whether ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109074 2022-11-23T03:14:26.9908691Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109075 2022-11-23T03:14:26.9909070Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9909236Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9909617Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9909784Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9910019Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp09ii1jtv 2022-11-23T03:14:26.9910267Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp09ii1jtv/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9910482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9910855Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9911022Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9911406Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9911583Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9911821Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnn7z3bpo 2022-11-23T03:14:26.9912069Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnn7z3bpo/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9912287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9912383Z ok (4.877s) 2022-11-23T03:14:26.9912389Z 2022-11-23T03:14:26.9912656Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9912760Z Ran 1 test in 4.878s 2022-11-23T03:14:26.9912765Z 2022-11-23T03:14:26.9912851Z OK 2022-11-23T03:14:26.9912856Z 2022-11-23T03:14:26.9912972Z Generating XML reports... 2022-11-23T03:14:26.9913425Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123030138.xml 2022-11-23T03:14:26.9913799Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9913966Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9914345Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9914534Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9914812Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsv9nmh2q 2022-11-23T03:14:26.9915066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsv9nmh2q/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9915381Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9915387Z 2022-11-23T03:14:26.9915487Z Running tests... 2022-11-23T03:14:26.9915753Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9915955Z test_ddp_comm_hook_sparse_gradients (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9916273Z Runs "test_sparse_gradients" unit test with DDP communication hook. We define a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109281 2022-11-23T03:14:26.9916480Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109282 2022-11-23T03:14:26.9916903Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9917071Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9917456Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9917637Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9917873Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnz8dfdih 2022-11-23T03:14:26.9918123Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnz8dfdih/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9918340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9918712Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9918877Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9919263Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9919438Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9919673Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq9pxyxc9 2022-11-23T03:14:26.9919925Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq9pxyxc9/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9920141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9920221Z ok (4.878s) 2022-11-23T03:14:26.9920241Z 2022-11-23T03:14:26.9920495Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9920597Z Ran 1 test in 4.879s 2022-11-23T03:14:26.9920603Z 2022-11-23T03:14:26.9920688Z OK 2022-11-23T03:14:26.9920694Z 2022-11-23T03:14:26.9920811Z Generating XML reports... 2022-11-23T03:14:26.9921267Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123030147.xml 2022-11-23T03:14:26.9921639Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9921803Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9922186Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9922362Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9922594Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6v5xtcdw 2022-11-23T03:14:26.9922842Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6v5xtcdw/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9923162Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9923220Z 2022-11-23T03:14:26.9923324Z Running tests... 2022-11-23T03:14:26.9923595Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9923791Z test_ddp_invalid_comm_hook_init (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9924106Z This unit test makes sure that register_comm_hook properly checks the format ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109554 2022-11-23T03:14:26.9924314Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109555 2022-11-23T03:14:26.9924693Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9924857Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9925240Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9925459Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9925688Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8uezxd2_ 2022-11-23T03:14:26.9925936Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8uezxd2_/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9926149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9926526Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9926691Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9927074Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9927252Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9927486Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmc_ecn8w 2022-11-23T03:14:26.9927861Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmc_ecn8w/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9928076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9928167Z ok (5.275s) 2022-11-23T03:14:26.9928173Z 2022-11-23T03:14:26.9928447Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9928547Z Ran 1 test in 5.275s 2022-11-23T03:14:26.9928553Z 2022-11-23T03:14:26.9928637Z OK 2022-11-23T03:14:26.9928642Z 2022-11-23T03:14:26.9928762Z Generating XML reports... 2022-11-23T03:14:26.9929204Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123030156.xml 2022-11-23T03:14:26.9929576Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9929739Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9930129Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9930314Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9930550Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpywfmlopy 2022-11-23T03:14:26.9930786Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpywfmlopy/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9931096Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9931102Z 2022-11-23T03:14:26.9931204Z Running tests... 2022-11-23T03:14:26.9931468Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9931671Z test_ddp_invalid_comm_hook_return_type (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9931996Z This test checks whether return annotation checked properly if defined. It also ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109761 2022-11-23T03:14:26.9932281Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109762 2022-11-23T03:14:26.9932657Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9932820Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9933204Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9933382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9933620Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnwkxi8i3 2022-11-23T03:14:26.9933866Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnwkxi8i3/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9934314Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9934487Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9934872Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9935051Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9935283Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxf4hme47 2022-11-23T03:14:26.9935530Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxf4hme47/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9935768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9935989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9936081Z ok (4.882s) 2022-11-23T03:14:26.9936088Z 2022-11-23T03:14:26.9936345Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9936454Z Ran 1 test in 4.883s 2022-11-23T03:14:26.9936460Z 2022-11-23T03:14:26.9936545Z OK 2022-11-23T03:14:26.9936550Z 2022-11-23T03:14:26.9936665Z Generating XML reports... 2022-11-23T03:14:26.9937114Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123030205.xml 2022-11-23T03:14:26.9937486Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9937653Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9938035Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9938215Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9938448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0iih9o8q 2022-11-23T03:14:26.9938704Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0iih9o8q/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9939019Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9939025Z 2022-11-23T03:14:26.9939127Z Running tests... 2022-11-23T03:14:26.9939393Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9939629Z test_find_unused_parameters_when_unused_parameters_empty (__main__.DistributedDataParallelTest) 2022-11-23T03:14:26.9939935Z An empty unused_parameters array does not imply find_unused_parameters = ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109972 2022-11-23T03:14:26.9940143Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109973 2022-11-23T03:14:26.9940517Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9940682Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9941148Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9941334Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9941569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmufhtjdr 2022-11-23T03:14:26.9941807Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmufhtjdr/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9942022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9942396Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9942562Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9942944Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9943169Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9943402Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_5oz4knx 2022-11-23T03:14:26.9943651Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_5oz4knx/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9943863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9944623Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T03:14:26.9944722Z ok (5.275s) 2022-11-23T03:14:26.9944728Z 2022-11-23T03:14:26.9945000Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9945105Z Ran 1 test in 5.275s 2022-11-23T03:14:26.9945111Z 2022-11-23T03:14:26.9945197Z OK 2022-11-23T03:14:26.9945202Z 2022-11-23T03:14:26.9945319Z Generating XML reports... 2022-11-23T03:14:26.9945771Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123030214.xml 2022-11-23T03:14:26.9946144Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9946307Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9946691Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9946871Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9947112Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa9nh_whm 2022-11-23T03:14:26.9947362Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa9nh_whm/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9947678Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9947684Z 2022-11-23T03:14:26.9947782Z Running tests... 2022-11-23T03:14:26.9948045Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9948375Z test_global_local_unused_params_grad (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110187 2022-11-23T03:14:26.9948570Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110188 2022-11-23T03:14:26.9948947Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9949119Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9949557Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9949736Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9949972Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr2kgdi7s 2022-11-23T03:14:26.9950225Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr2kgdi7s/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9950443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9950815Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9950983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9951407Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9951589Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9951828Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcestmyhf 2022-11-23T03:14:26.9952078Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcestmyhf/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9952294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9952385Z ok (5.191s) 2022-11-23T03:14:26.9952390Z 2022-11-23T03:14:26.9952662Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9952764Z Ran 1 test in 5.191s 2022-11-23T03:14:26.9952770Z 2022-11-23T03:14:26.9952856Z OK 2022-11-23T03:14:26.9952861Z 2022-11-23T03:14:26.9952978Z Generating XML reports... 2022-11-23T03:14:26.9953429Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123030223.xml 2022-11-23T03:14:26.9953808Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9953960Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9954343Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9954519Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9954754Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp79gu744e 2022-11-23T03:14:26.9955001Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp79gu744e/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9955312Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9955318Z 2022-11-23T03:14:26.9955416Z Running tests... 2022-11-23T03:14:26.9955682Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9956032Z test_global_local_unused_params_grad_with_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110402 2022-11-23T03:14:26.9956237Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110403 2022-11-23T03:14:26.9956610Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9956779Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9957161Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9957342Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9957578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpes5xfzop 2022-11-23T03:14:26.9957832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpes5xfzop/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9958109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9958483Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9958646Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9959029Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9959213Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9959448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpderh8t74 2022-11-23T03:14:26.9959683Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpderh8t74/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9959894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9960034Z ok (5.067s) 2022-11-23T03:14:26.9960041Z 2022-11-23T03:14:26.9960313Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9960413Z Ran 1 test in 5.067s 2022-11-23T03:14:26.9960420Z 2022-11-23T03:14:26.9960509Z OK 2022-11-23T03:14:26.9960515Z 2022-11-23T03:14:26.9960635Z Generating XML reports... 2022-11-23T03:14:26.9961086Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123030232.xml 2022-11-23T03:14:26.9961458Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9961621Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9962001Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9962181Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9962423Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7hvjpyuc 2022-11-23T03:14:26.9962679Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7hvjpyuc/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9962991Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9962997Z 2022-11-23T03:14:26.9963098Z Running tests... 2022-11-23T03:14:26.9963367Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9963714Z test_global_local_unused_params_grad_with_static_graph (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110617 2022-11-23T03:14:26.9963921Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110618 2022-11-23T03:14:26.9964298Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9964466Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9964848Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9965014Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9965250Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcolveiy0 2022-11-23T03:14:26.9965497Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcolveiy0/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9965710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9966081Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9966245Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9966633Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9966873Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9967106Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw7r3_amt 2022-11-23T03:14:26.9967352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw7r3_amt/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9967567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9968586Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T03:14:26.9968688Z warnings.warn( 2022-11-23T03:14:26.9969680Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T03:14:26.9969786Z warnings.warn( 2022-11-23T03:14:26.9969880Z ok (5.171s) 2022-11-23T03:14:26.9969887Z 2022-11-23T03:14:26.9970163Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9970266Z Ran 1 test in 5.171s 2022-11-23T03:14:26.9970272Z 2022-11-23T03:14:26.9970358Z OK 2022-11-23T03:14:26.9970364Z 2022-11-23T03:14:26.9970481Z Generating XML reports... 2022-11-23T03:14:26.9970928Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123030242.xml 2022-11-23T03:14:26.9971305Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9971472Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9971852Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9972031Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9972267Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprkhtmhe2 2022-11-23T03:14:26.9972503Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprkhtmhe2/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9972812Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9972817Z 2022-11-23T03:14:26.9972918Z Running tests... 2022-11-23T03:14:26.9973182Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9973537Z test_gloo_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110832 2022-11-23T03:14:26.9973747Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110833 2022-11-23T03:14:26.9974116Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9974283Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9974668Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9974845Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9975079Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6r9wjsxb 2022-11-23T03:14:26.9975324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6r9wjsxb/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9975604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9975978Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9976145Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9976528Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9976706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9976939Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqu6ueso8 2022-11-23T03:14:26.9977185Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqu6ueso8/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9977403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9977669Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9977897Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9977975Z ok (7.601s) 2022-11-23T03:14:26.9977980Z 2022-11-23T03:14:26.9978253Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9978355Z Ran 1 test in 7.601s 2022-11-23T03:14:26.9978360Z 2022-11-23T03:14:26.9978448Z OK 2022-11-23T03:14:26.9978453Z 2022-11-23T03:14:26.9978568Z Generating XML reports... 2022-11-23T03:14:26.9979018Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123030251.xml 2022-11-23T03:14:26.9979389Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9979556Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9979941Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9980119Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9980350Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_x4q6o30 2022-11-23T03:14:26.9980596Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_x4q6o30/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9980911Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9980917Z 2022-11-23T03:14:26.9981016Z Running tests... 2022-11-23T03:14:26.9981284Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9981637Z test_gloo_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111049 2022-11-23T03:14:26.9981845Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111050 2022-11-23T03:14:26.9982221Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9982386Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9982771Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9982946Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9983180Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9lh1k8ll 2022-11-23T03:14:26.9983414Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9lh1k8ll/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9983630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9984003Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9984231Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9984614Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9984794Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9985027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbelu8nea 2022-11-23T03:14:26.9985277Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbelu8nea/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9985491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9985710Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9985928Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:26.9986019Z ok (7.283s) 2022-11-23T03:14:26.9986025Z 2022-11-23T03:14:26.9986342Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9986446Z Ran 1 test in 7.283s 2022-11-23T03:14:26.9986452Z 2022-11-23T03:14:26.9986536Z OK 2022-11-23T03:14:26.9986542Z 2022-11-23T03:14:26.9986657Z Generating XML reports... 2022-11-23T03:14:26.9987104Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123030302.xml 2022-11-23T03:14:26.9987476Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9987640Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9988023Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9988201Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9988424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphllwnpau 2022-11-23T03:14:26.9988682Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphllwnpau/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9988994Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9989000Z 2022-11-23T03:14:26.9989099Z Running tests... 2022-11-23T03:14:26.9989364Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9989679Z test_gloo_backend_2gpu_module (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111266 2022-11-23T03:14:26.9989890Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111267 2022-11-23T03:14:26.9990265Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9990434Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9990817Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9990997Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9991231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0fm25t_a 2022-11-23T03:14:26.9991477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0fm25t_a/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9991691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:26.9992065Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9992229Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9992609Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9992791Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9993092Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnbgvtuw5 2022-11-23T03:14:26.9993343Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnbgvtuw5/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9993556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9993693Z skip: Need at least 4 CUDA devices (4.682s) 2022-11-23T03:14:26.9993700Z 2022-11-23T03:14:26.9993958Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9994064Z Ran 1 test in 4.682s 2022-11-23T03:14:26.9994070Z 2022-11-23T03:14:26.9994169Z OK (skipped=1) 2022-11-23T03:14:26.9994175Z 2022-11-23T03:14:26.9994293Z Generating XML reports... 2022-11-23T03:14:26.9994743Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123030314.xml 2022-11-23T03:14:26.9995167Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9995340Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9995727Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9995908Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9996140Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpggci0lfn 2022-11-23T03:14:26.9996391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpggci0lfn/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9996705Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:26.9996711Z 2022-11-23T03:14:26.9996811Z Running tests... 2022-11-23T03:14:26.9997076Z ---------------------------------------------------------------------- 2022-11-23T03:14:26.9997392Z test_gloo_backend_4gpu_module (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111467 2022-11-23T03:14:26.9997605Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111468 2022-11-23T03:14:26.9997981Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9998149Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:26.9998529Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:26.9998706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:26.9998944Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb2ptmdxx 2022-11-23T03:14:26.9999196Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb2ptmdxx/_remote_module_non_scriptable.py 2022-11-23T03:14:26.9999400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:26.9999773Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:26.9999939Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0000322Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0000500Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0000733Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp70kbvkyy 2022-11-23T03:14:27.0000980Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp70kbvkyy/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0001193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0001332Z skip: Need at least 8 CUDA devices (4.687s) 2022-11-23T03:14:27.0001401Z 2022-11-23T03:14:27.0001679Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0001781Z Ran 1 test in 4.687s 2022-11-23T03:14:27.0001787Z 2022-11-23T03:14:27.0001884Z OK (skipped=1) 2022-11-23T03:14:27.0001890Z 2022-11-23T03:14:27.0002008Z Generating XML reports... 2022-11-23T03:14:27.0002458Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123030322.xml 2022-11-23T03:14:27.0002825Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0002988Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0003376Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0003553Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0003847Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppytrya2_ 2022-11-23T03:14:27.0004101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppytrya2_/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0004420Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0004426Z 2022-11-23T03:14:27.0004531Z Running tests... 2022-11-23T03:14:27.0004784Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0005102Z test_gloo_backend_cpu_module (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111668 2022-11-23T03:14:27.0005311Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111669 2022-11-23T03:14:27.0005683Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0005851Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0006241Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0006420Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0006654Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp27yzx0ag 2022-11-23T03:14:27.0006899Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp27yzx0ag/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0007272Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0007433Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0007940Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0008123Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0008363Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6mva9vc4 2022-11-23T03:14:27.0008613Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6mva9vc4/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0008825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0009034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0009254Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:27.0009472Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:27.0009562Z ok (4.880s) 2022-11-23T03:14:27.0009568Z 2022-11-23T03:14:27.0009842Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0009930Z Ran 1 test in 4.880s 2022-11-23T03:14:27.0009951Z 2022-11-23T03:14:27.0010023Z OK 2022-11-23T03:14:27.0010029Z 2022-11-23T03:14:27.0010225Z Generating XML reports... 2022-11-23T03:14:27.0010678Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123030331.xml 2022-11-23T03:14:27.0011049Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0011215Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0011595Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0011776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0012014Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8kxnxxiv 2022-11-23T03:14:27.0012262Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8kxnxxiv/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0012626Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0012637Z 2022-11-23T03:14:27.0012739Z Running tests... 2022-11-23T03:14:27.0013009Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0013339Z test_gloo_backend_cpu_module_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111941 2022-11-23T03:14:27.0013544Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111942 2022-11-23T03:14:27.0013922Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0014086Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0014469Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0014647Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0014889Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwuz3pim1 2022-11-23T03:14:27.0015138Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwuz3pim1/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0015509Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0015671Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0016038Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0016214Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0016446Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4be4hxlx 2022-11-23T03:14:27.0016694Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4be4hxlx/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0016912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0017128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0017342Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:27.0017561Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:27.0017651Z ok (4.987s) 2022-11-23T03:14:27.0017657Z 2022-11-23T03:14:27.0017925Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0018026Z Ran 1 test in 4.987s 2022-11-23T03:14:27.0018032Z 2022-11-23T03:14:27.0018116Z OK 2022-11-23T03:14:27.0018121Z 2022-11-23T03:14:27.0018235Z Generating XML reports... 2022-11-23T03:14:27.0018690Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123030340.xml 2022-11-23T03:14:27.0019064Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0019293Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0019678Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0019857Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0020093Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyj6p24wk 2022-11-23T03:14:27.0020339Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyj6p24wk/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0020652Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0020658Z 2022-11-23T03:14:27.0020745Z Running tests... 2022-11-23T03:14:27.0021015Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0021193Z test_ignored_output (__main__.DistributedDataParallelTest) 2022-11-23T03:14:27.0021537Z Test that the output of a model can be ignored and that there is no ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112214 2022-11-23T03:14:27.0021748Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112215 2022-11-23T03:14:27.0022128Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0022295Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0022680Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0022859Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0023095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpder_gdr_ 2022-11-23T03:14:27.0023341Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpder_gdr_/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0023717Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0023881Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0024262Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0024440Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0024675Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgp3d9hzw 2022-11-23T03:14:27.0024927Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgp3d9hzw/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0025140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0025356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0025578Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:27.0025795Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:27.0025873Z ok (5.885s) 2022-11-23T03:14:27.0025892Z 2022-11-23T03:14:27.0026148Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0026248Z Ran 1 test in 5.886s 2022-11-23T03:14:27.0026254Z 2022-11-23T03:14:27.0026339Z OK 2022-11-23T03:14:27.0026345Z 2022-11-23T03:14:27.0026458Z Generating XML reports... 2022-11-23T03:14:27.0026910Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123030349.xml 2022-11-23T03:14:27.0027281Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0027445Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0027829Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0028064Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0028300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpikxezixa 2022-11-23T03:14:27.0028550Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpikxezixa/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0028907Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0028914Z 2022-11-23T03:14:27.0029034Z Running tests... 2022-11-23T03:14:27.0029354Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0029608Z test_ignored_output_with_unused_parameters (__main__.DistributedDataParallelTest) 2022-11-23T03:14:27.0029956Z Test that the output of a model can be ignored and that there is no ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112487 2022-11-23T03:14:27.0030258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112488 2022-11-23T03:14:27.0030711Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0030911Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0031368Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0031584Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0031853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuyfen5rw 2022-11-23T03:14:27.0032155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuyfen5rw/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0032602Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0032806Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0033265Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0033473Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0033751Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpryfczl_r 2022-11-23T03:14:27.0034053Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpryfczl_r/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0034313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0034569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0034681Z ok (5.884s) 2022-11-23T03:14:27.0034688Z 2022-11-23T03:14:27.0035007Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0035134Z Ran 1 test in 5.885s 2022-11-23T03:14:27.0035144Z 2022-11-23T03:14:27.0035246Z OK 2022-11-23T03:14:27.0035253Z 2022-11-23T03:14:27.0035392Z Generating XML reports... 2022-11-23T03:14:27.0035931Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123030359.xml 2022-11-23T03:14:27.0036374Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0036570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0037032Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0037245Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0037528Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdqcsq357 2022-11-23T03:14:27.0037815Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdqcsq357/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0038265Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0038272Z 2022-11-23T03:14:27.0038393Z Running tests... 2022-11-23T03:14:27.0038711Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0039092Z test_ignored_sharded_tensor (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112760 2022-11-23T03:14:27.0039342Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112761 2022-11-23T03:14:27.0039785Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0039988Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0040445Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0040716Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0040999Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv9o20iof 2022-11-23T03:14:27.0041294Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv9o20iof/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0041554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0042003Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0042198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0042654Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0042865Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0043152Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7gcr6o3o 2022-11-23T03:14:27.0043452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7gcr6o3o/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0043708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0043982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:14:27.0044248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:14:27.0044709Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:27.0045189Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:27.0045301Z ok (5.177s) 2022-11-23T03:14:27.0045308Z 2022-11-23T03:14:27.0045627Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0045753Z Ran 1 test in 5.177s 2022-11-23T03:14:27.0045760Z 2022-11-23T03:14:27.0045858Z OK 2022-11-23T03:14:27.0045864Z 2022-11-23T03:14:27.0046008Z Generating XML reports... 2022-11-23T03:14:27.0046503Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123030409.xml 2022-11-23T03:14:27.0046874Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0047038Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0047422Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0047614Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0047931Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp494xhp8x 2022-11-23T03:14:27.0048181Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp494xhp8x/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0048568Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0048575Z 2022-11-23T03:14:27.0048676Z Running tests... 2022-11-23T03:14:27.0048940Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0049252Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112967 2022-11-23T03:14:27.0049458Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112968 2022-11-23T03:14:27.0049832Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0049998Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0050385Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0050613Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0050837Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvn0o_yiv 2022-11-23T03:14:27.0051085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvn0o_yiv/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0051295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0051822Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:14:27.0052345Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:14:27.0052881Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:14:27.0053398Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:14:27.0053918Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:14:27.0054434Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:14:27.0054814Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0054977Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0055357Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0055590Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0055827Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxtwehnz6 2022-11-23T03:14:27.0056077Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxtwehnz6/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0056280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0056808Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:14:27.0057358Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:14:27.0057873Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:14:27.0058378Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:14:27.0058893Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:14:27.0059405Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:14:27.0059499Z ok (4.680s) 2022-11-23T03:14:27.0059505Z 2022-11-23T03:14:27.0059779Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0059878Z Ran 1 test in 4.680s 2022-11-23T03:14:27.0059885Z 2022-11-23T03:14:27.0059969Z OK 2022-11-23T03:14:27.0059975Z 2022-11-23T03:14:27.0060091Z Generating XML reports... 2022-11-23T03:14:27.0060551Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123030418.xml 2022-11-23T03:14:27.0060927Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0061091Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0061474Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0061640Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0061874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpje5la5g_ 2022-11-23T03:14:27.0062121Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpje5la5g_/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0062435Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0062491Z 2022-11-23T03:14:27.0062596Z Running tests... 2022-11-23T03:14:27.0062868Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0063187Z test_save_load_checkpoint (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113168 2022-11-23T03:14:27.0063394Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113169 2022-11-23T03:14:27.0063763Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0063927Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0064311Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0064489Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0064769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnnoscpnh 2022-11-23T03:14:27.0065028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnnoscpnh/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0065248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0065621Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0065786Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0066167Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0066349Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0066586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdowwyo2o 2022-11-23T03:14:27.0066836Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdowwyo2o/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0067057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0067271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:14:27.0067497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:14:27.0067892Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:27.0068286Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:14:27.0068506Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:27.0068731Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:27.0068946Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:27.0069164Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:27.0069255Z ok (7.486s) 2022-11-23T03:14:27.0069261Z 2022-11-23T03:14:27.0069531Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0069633Z Ran 1 test in 7.486s 2022-11-23T03:14:27.0069639Z 2022-11-23T03:14:27.0069724Z OK 2022-11-23T03:14:27.0069729Z 2022-11-23T03:14:27.0069845Z Generating XML reports... 2022-11-23T03:14:27.0070294Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123030427.xml 2022-11-23T03:14:27.0070665Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0070830Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0071215Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0071473Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0071709Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp__0m9xgt 2022-11-23T03:14:27.0071954Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp__0m9xgt/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0072271Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0072277Z 2022-11-23T03:14:27.0072377Z Running tests... 2022-11-23T03:14:27.0072629Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0072936Z test_sparse_gradients (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113385 2022-11-23T03:14:27.0073143Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113386 2022-11-23T03:14:27.0073561Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0073736Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0074123Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0074302Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0074536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprx477ehs 2022-11-23T03:14:27.0074783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprx477ehs/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0075000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0075367Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0075530Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0075916Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0076094Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0076330Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkonb1tae 2022-11-23T03:14:27.0076582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkonb1tae/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0076801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0076895Z ok (5.179s) 2022-11-23T03:14:27.0076900Z 2022-11-23T03:14:27.0077164Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0077265Z Ran 1 test in 5.179s 2022-11-23T03:14:27.0077271Z 2022-11-23T03:14:27.0077358Z OK 2022-11-23T03:14:27.0077364Z 2022-11-23T03:14:27.0077467Z Generating XML reports... 2022-11-23T03:14:27.0077918Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123030439.xml 2022-11-23T03:14:27.0078289Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0078453Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0078832Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0079009Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0079242Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpou_3yii5 2022-11-23T03:14:27.0079486Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpou_3yii5/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0079807Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0079813Z 2022-11-23T03:14:27.0079975Z Running tests... 2022-11-23T03:14:27.0080245Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0080565Z test_sparse_gradients_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113658 2022-11-23T03:14:27.0080773Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113659 2022-11-23T03:14:27.0081146Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0081312Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0081692Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0081873Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0082110Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpebpd5i69 2022-11-23T03:14:27.0082402Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpebpd5i69/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0082621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0082996Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0083162Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0083530Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0083707Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0083940Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpceehmh2p 2022-11-23T03:14:27.0084187Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpceehmh2p/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0084409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0084502Z ok (4.765s) 2022-11-23T03:14:27.0084508Z 2022-11-23T03:14:27.0084776Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0084879Z Ran 1 test in 4.765s 2022-11-23T03:14:27.0084884Z 2022-11-23T03:14:27.0084969Z OK 2022-11-23T03:14:27.0084977Z 2022-11-23T03:14:27.0085091Z Generating XML reports... 2022-11-23T03:14:27.0085538Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123030448.xml 2022-11-23T03:14:27.0085907Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0086070Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0086449Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0086631Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0086868Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe06wcx1o 2022-11-23T03:14:27.0087117Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe06wcx1o/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0087430Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0087436Z 2022-11-23T03:14:27.0087536Z Running tests... 2022-11-23T03:14:27.0087876Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0088197Z test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113931 2022-11-23T03:14:27.0088403Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113932 2022-11-23T03:14:27.0088766Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0089005Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0089393Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0089575Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0089811Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsmzmeo7r 2022-11-23T03:14:27.0090062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsmzmeo7r/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0090276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0090643Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0090812Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0091244Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0091424Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0091663Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd5nb_m_d 2022-11-23T03:14:27.0091909Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd5nb_m_d/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0092125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0092342Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:27.0092561Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:27.0092654Z ok (8.240s) 2022-11-23T03:14:27.0092661Z 2022-11-23T03:14:27.0092931Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0093042Z Ran 1 test in 8.240s 2022-11-23T03:14:27.0093048Z 2022-11-23T03:14:27.0093132Z OK 2022-11-23T03:14:27.0093137Z 2022-11-23T03:14:27.0093251Z Generating XML reports... 2022-11-23T03:14:27.0093686Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123030457.xml 2022-11-23T03:14:27.0094056Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0094223Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0094602Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0094785Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0095022Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo5elhhd6 2022-11-23T03:14:27.0095274Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo5elhhd6/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0095594Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0095599Z 2022-11-23T03:14:27.0095701Z Running tests... 2022-11-23T03:14:27.0095968Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0096293Z test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114187 2022-11-23T03:14:27.0096507Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114188 2022-11-23T03:14:27.0096878Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0097045Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0097427Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0097668Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0097904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpozhsyxnj 2022-11-23T03:14:27.0098154Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpozhsyxnj/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0098366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0098747Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0098911Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0099296Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0099462Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0099738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp31tbxtwz 2022-11-23T03:14:27.0099990Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp31tbxtwz/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0100203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0100424Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:27.0100641Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:14:27.0100733Z ok (5.559s) 2022-11-23T03:14:27.0100738Z 2022-11-23T03:14:27.0101009Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0101118Z Ran 1 test in 5.560s 2022-11-23T03:14:27.0101124Z 2022-11-23T03:14:27.0101211Z OK 2022-11-23T03:14:27.0101217Z 2022-11-23T03:14:27.0101331Z Generating XML reports... 2022-11-23T03:14:27.0101785Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221123030509.xml 2022-11-23T03:14:27.0102160Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0102327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0102709Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0102888Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0103126Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplyn5_xez 2022-11-23T03:14:27.0103372Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplyn5_xez/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0103688Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0103694Z 2022-11-23T03:14:27.0103793Z Running tests... 2022-11-23T03:14:27.0104066Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0104420Z test_allgather_coalesced (__main__.GlooProcessGroupWithDispatchedCollectivesTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114402 2022-11-23T03:14:27.0104798Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0104962Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0105345Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0105523Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0105752Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_6iky982 2022-11-23T03:14:27.0105999Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_6iky982/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0106289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0106517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:14:27.0106922Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:14:27.0107682Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T03:14:27.0107788Z warnings.warn( 2022-11-23T03:14:27.0107880Z ok (5.175s) 2022-11-23T03:14:27.0107886Z 2022-11-23T03:14:27.0108155Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0108256Z Ran 1 test in 5.175s 2022-11-23T03:14:27.0108262Z 2022-11-23T03:14:27.0108351Z OK 2022-11-23T03:14:27.0108402Z 2022-11-23T03:14:27.0108519Z Generating XML reports... 2022-11-23T03:14:27.0109061Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20221123030519.xml 2022-11-23T03:14:27.0109432Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0109602Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0109984Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0110161Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0110393Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3rb3s_nc 2022-11-23T03:14:27.0110639Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3rb3s_nc/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0110944Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0110963Z 2022-11-23T03:14:27.0111050Z Running tests... 2022-11-23T03:14:27.0111316Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0111685Z test_allreduce_coalesced (__main__.GlooProcessGroupWithDispatchedCollectivesTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114539 2022-11-23T03:14:27.0112055Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0112216Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0112598Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0112782Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0113023Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz1bq576k 2022-11-23T03:14:27.0113271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz1bq576k/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0113486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0113713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:14:27.0114108Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:14:27.0114851Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T03:14:27.0114955Z warnings.warn( 2022-11-23T03:14:27.0115051Z ok (4.576s) 2022-11-23T03:14:27.0115119Z 2022-11-23T03:14:27.0115394Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0115494Z Ran 1 test in 4.577s 2022-11-23T03:14:27.0115500Z 2022-11-23T03:14:27.0115585Z OK 2022-11-23T03:14:27.0115590Z 2022-11-23T03:14:27.0115707Z Generating XML reports... 2022-11-23T03:14:27.0116251Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20221123030528.xml 2022-11-23T03:14:27.0116623Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0116791Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0117174Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0117339Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0117620Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp85xeyx28 2022-11-23T03:14:27.0117872Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp85xeyx28/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0118191Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0118197Z 2022-11-23T03:14:27.0118299Z Running tests... 2022-11-23T03:14:27.0118568Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0118921Z test_collectives (__main__.GlooProcessGroupWithDispatchedCollectivesTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114676 2022-11-23T03:14:27.0119291Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0119458Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0119843Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0120029Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0120267Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkpvqt46r 2022-11-23T03:14:27.0120517Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkpvqt46r/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0120730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0120955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:14:27.0121353Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:14:27.0121446Z ok (4.682s) 2022-11-23T03:14:27.0121452Z 2022-11-23T03:14:27.0121718Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0121820Z Ran 1 test in 4.682s 2022-11-23T03:14:27.0121829Z 2022-11-23T03:14:27.0121916Z OK 2022-11-23T03:14:27.0121922Z 2022-11-23T03:14:27.0122035Z Generating XML reports... 2022-11-23T03:14:27.0122571Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20221123030536.xml 2022-11-23T03:14:27.0122930Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0123099Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0123484Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0123662Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0123896Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph5kwk6sa 2022-11-23T03:14:27.0124148Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph5kwk6sa/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0124531Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0124537Z 2022-11-23T03:14:27.0124641Z Running tests... 2022-11-23T03:14:27.0124908Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0125272Z test_monitored_barrier (__main__.GlooProcessGroupWithDispatchedCollectivesTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114813 2022-11-23T03:14:27.0125643Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0125808Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0126189Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0126369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0126652Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgybt5kjn 2022-11-23T03:14:27.0126904Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgybt5kjn/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0127121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0127348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:14:27.0127881Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:14:27.0127979Z ok (5.121s) 2022-11-23T03:14:27.0127985Z 2022-11-23T03:14:27.0128254Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0128356Z Ran 1 test in 5.121s 2022-11-23T03:14:27.0128361Z 2022-11-23T03:14:27.0128433Z OK 2022-11-23T03:14:27.0128452Z 2022-11-23T03:14:27.0128554Z Generating XML reports... 2022-11-23T03:14:27.0129097Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20221123030545.xml 2022-11-23T03:14:27.0129467Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0129634Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0130016Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0130196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0130434Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptm_9tp6a 2022-11-23T03:14:27.0130679Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptm_9tp6a/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0130992Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0131001Z 2022-11-23T03:14:27.0131104Z Running tests... 2022-11-23T03:14:27.0131372Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0131671Z test_allgather_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114950 2022-11-23T03:14:27.0131877Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114951 2022-11-23T03:14:27.0132085Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 114952 2022-11-23T03:14:27.0132289Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 114953 2022-11-23T03:14:27.0132663Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0132828Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0133211Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0133459Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0133696Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk66p4wgq 2022-11-23T03:14:27.0133944Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk66p4wgq/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0134146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0134524Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0134687Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0135069Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0135249Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0135549Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps_zzkqou 2022-11-23T03:14:27.0135796Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps_zzkqou/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0136175Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0136342Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0136722Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0136905Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0137140Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy16dvbma 2022-11-23T03:14:27.0137387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy16dvbma/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0137604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0137825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0138197Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0138365Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0138746Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0138925Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0139160Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9dhedqr_ 2022-11-23T03:14:27.0139406Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9dhedqr_/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0139621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0139706Z ok (5.226s) 2022-11-23T03:14:27.0139712Z 2022-11-23T03:14:27.0139980Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0140082Z Ran 1 test in 5.227s 2022-11-23T03:14:27.0140088Z 2022-11-23T03:14:27.0140173Z OK 2022-11-23T03:14:27.0140179Z 2022-11-23T03:14:27.0140294Z Generating XML reports... 2022-11-23T03:14:27.0140720Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123030554.xml 2022-11-23T03:14:27.0141090Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0141260Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0141642Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0141828Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0142135Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp4d0luzn 2022-11-23T03:14:27.0142384Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp4d0luzn/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0142695Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0142701Z 2022-11-23T03:14:27.0142803Z Running tests... 2022-11-23T03:14:27.0143070Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0143369Z test_allgather_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115297 2022-11-23T03:14:27.0143575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115298 2022-11-23T03:14:27.0143780Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 115299 2022-11-23T03:14:27.0143987Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 115300 2022-11-23T03:14:27.0144414Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0144581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0144953Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0145132Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0145366Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp194b9n7j 2022-11-23T03:14:27.0145616Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp194b9n7j/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0145835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0146217Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0146386Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0146768Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0146945Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0147182Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp43s6eci8 2022-11-23T03:14:27.0147428Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp43s6eci8/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0147801Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0147966Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0148350Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0148534Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0148776Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphepl90fj 2022-11-23T03:14:27.0149025Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphepl90fj/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0149397Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0149558Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0149937Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0150113Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0150347Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgrqdx40r 2022-11-23T03:14:27.0150583Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgrqdx40r/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0150857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0151070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0151282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0151375Z ok (6.175s) 2022-11-23T03:14:27.0151381Z 2022-11-23T03:14:27.0151650Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0151751Z Ran 1 test in 6.175s 2022-11-23T03:14:27.0151757Z 2022-11-23T03:14:27.0151840Z OK 2022-11-23T03:14:27.0151847Z 2022-11-23T03:14:27.0151961Z Generating XML reports... 2022-11-23T03:14:27.0152386Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123030604.xml 2022-11-23T03:14:27.0152758Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0152969Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0153358Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0153541Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0153775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfnke6dh6 2022-11-23T03:14:27.0154023Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfnke6dh6/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0154337Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0154343Z 2022-11-23T03:14:27.0154448Z Running tests... 2022-11-23T03:14:27.0154714Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0155015Z test_allgather_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115644 2022-11-23T03:14:27.0155229Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115645 2022-11-23T03:14:27.0155421Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 115646 2022-11-23T03:14:27.0155629Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 115647 2022-11-23T03:14:27.0156000Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0156163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0156549Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0156729Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0156962Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_e4wlejl 2022-11-23T03:14:27.0157219Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_e4wlejl/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0157433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0157805Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0157969Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0158353Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0158533Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0158769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl3c263df 2022-11-23T03:14:27.0159015Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl3c263df/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0159230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0159664Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0159827Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0160210Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0160388Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0160622Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8619wpto 2022-11-23T03:14:27.0160869Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8619wpto/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0161069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0161483Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0161655Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0162040Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0162213Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0162451Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt43t2pi2 2022-11-23T03:14:27.0162705Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt43t2pi2/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0162917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0163011Z ok (4.986s) 2022-11-23T03:14:27.0163017Z 2022-11-23T03:14:27.0163287Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0163389Z Ran 1 test in 4.987s 2022-11-23T03:14:27.0163395Z 2022-11-23T03:14:27.0163486Z OK 2022-11-23T03:14:27.0163494Z 2022-11-23T03:14:27.0163615Z Generating XML reports... 2022-11-23T03:14:27.0164041Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123030614.xml 2022-11-23T03:14:27.0164410Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0164574Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0164956Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0165133Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0165369Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpimb8za1l 2022-11-23T03:14:27.0165618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpimb8za1l/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0165936Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0165945Z 2022-11-23T03:14:27.0166032Z Running tests... 2022-11-23T03:14:27.0166299Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0166605Z test_allgather_coalesced_async (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115991 2022-11-23T03:14:27.0166813Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115992 2022-11-23T03:14:27.0167020Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 115993 2022-11-23T03:14:27.0167228Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 115994 2022-11-23T03:14:27.0167604Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0167839Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0168305Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0168485Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0168720Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpihtkqbt4 2022-11-23T03:14:27.0168970Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpihtkqbt4/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0169184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0169556Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0169721Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0170101Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0170327Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0170570Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgxn95jrj 2022-11-23T03:14:27.0170822Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgxn95jrj/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0171037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0171409Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0171574Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0171941Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0172117Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0172354Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppd10nfzw 2022-11-23T03:14:27.0172609Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppd10nfzw/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0172827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0173198Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0173363Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0173745Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0173924Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0174158Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpai5yd8el 2022-11-23T03:14:27.0174411Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpai5yd8el/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0174633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0174862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:14:27.0175089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:14:27.0175317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T03:14:27.0175713Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T03:14:27.0176103Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T03:14:27.0176325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T03:14:27.0176721Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T03:14:27.0177177Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T03:14:27.0177920Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T03:14:27.0178024Z warnings.warn( 2022-11-23T03:14:27.0178763Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T03:14:27.0178867Z warnings.warn( 2022-11-23T03:14:27.0179653Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T03:14:27.0179758Z warnings.warn( 2022-11-23T03:14:27.0180495Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T03:14:27.0180596Z warnings.warn( 2022-11-23T03:14:27.0180675Z ok (5.194s) 2022-11-23T03:14:27.0180693Z 2022-11-23T03:14:27.0180948Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0181048Z Ran 1 test in 5.194s 2022-11-23T03:14:27.0181054Z 2022-11-23T03:14:27.0181137Z OK 2022-11-23T03:14:27.0181142Z 2022-11-23T03:14:27.0181257Z Generating XML reports... 2022-11-23T03:14:27.0181690Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123030623.xml 2022-11-23T03:14:27.0182064Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0182229Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0182616Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0182798Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0183034Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj2muccx5 2022-11-23T03:14:27.0183285Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj2muccx5/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0183602Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0183608Z 2022-11-23T03:14:27.0183711Z Running tests... 2022-11-23T03:14:27.0183979Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0184286Z test_allgather_coalesced_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116338 2022-11-23T03:14:27.0184494Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116339 2022-11-23T03:14:27.0184703Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 116340 2022-11-23T03:14:27.0184913Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 116341 2022-11-23T03:14:27.0185285Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0185452Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0185823Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0186064Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0186300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplinfy7s4 2022-11-23T03:14:27.0186549Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplinfy7s4/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0186762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0187138Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0187307Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0187689Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0187871Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0188148Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpei9h3c10 2022-11-23T03:14:27.0188400Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpei9h3c10/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0188612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0188986Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0189155Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0189534Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0189709Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0189942Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0_bhlcu7 2022-11-23T03:14:27.0190189Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0_bhlcu7/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0190566Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0190730Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0191109Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0191293Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0191515Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpckr61ev3 2022-11-23T03:14:27.0191762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpckr61ev3/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0191979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0192194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0192952Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T03:14:27.0193055Z warnings.warn( 2022-11-23T03:14:27.0193797Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T03:14:27.0193900Z warnings.warn( 2022-11-23T03:14:27.0194640Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T03:14:27.0194803Z warnings.warn( 2022-11-23T03:14:27.0195546Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T03:14:27.0195646Z warnings.warn( 2022-11-23T03:14:27.0195742Z ok (5.111s) 2022-11-23T03:14:27.0195748Z 2022-11-23T03:14:27.0196019Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0196120Z Ran 1 test in 5.111s 2022-11-23T03:14:27.0196125Z 2022-11-23T03:14:27.0196207Z OK 2022-11-23T03:14:27.0196212Z 2022-11-23T03:14:27.0196331Z Generating XML reports... 2022-11-23T03:14:27.0196758Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123030632.xml 2022-11-23T03:14:27.0197188Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0197358Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0197742Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0197922Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0198157Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd3cyry9r 2022-11-23T03:14:27.0198410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd3cyry9r/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0198728Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0198735Z 2022-11-23T03:14:27.0198836Z Running tests... 2022-11-23T03:14:27.0199089Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0199403Z test_allgather_noncontiguous_input (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116685 2022-11-23T03:14:27.0199614Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116686 2022-11-23T03:14:27.0199825Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 116687 2022-11-23T03:14:27.0200029Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 116688 2022-11-23T03:14:27.0200401Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0200568Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0200954Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0201131Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0201366Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpefsx5iec 2022-11-23T03:14:27.0201620Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpefsx5iec/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0201991Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0202159Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0202539Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0202717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0202954Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps3gdvw9t 2022-11-23T03:14:27.0203204Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps3gdvw9t/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0203576Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0203821Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0204210Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0204389Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0204623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvkhd06s_ 2022-11-23T03:14:27.0204859Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvkhd06s_/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0205072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0205287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0205497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0205909Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0206083Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0206466Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0206644Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0206881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplh7uu8g9 2022-11-23T03:14:27.0207129Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplh7uu8g9/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0207342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0207435Z ok (4.977s) 2022-11-23T03:14:27.0207443Z 2022-11-23T03:14:27.0207855Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0207957Z Ran 1 test in 4.977s 2022-11-23T03:14:27.0207967Z 2022-11-23T03:14:27.0208058Z OK 2022-11-23T03:14:27.0208063Z 2022-11-23T03:14:27.0208178Z Generating XML reports... 2022-11-23T03:14:27.0208604Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123030641.xml 2022-11-23T03:14:27.0208978Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0209141Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0209524Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0209703Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0209925Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpti0aav_0 2022-11-23T03:14:27.0210174Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpti0aav_0/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0210494Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0210500Z 2022-11-23T03:14:27.0210600Z Running tests... 2022-11-23T03:14:27.0210863Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0211156Z test_allgather_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117032 2022-11-23T03:14:27.0211363Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117033 2022-11-23T03:14:27.0211567Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 117034 2022-11-23T03:14:27.0211773Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 117035 2022-11-23T03:14:27.0212149Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0212314Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0212776Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0212954Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0213187Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptd4_1ntv 2022-11-23T03:14:27.0213437Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptd4_1ntv/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0213651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0214022Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0214186Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0214566Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0214798Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0215039Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqf0cr66q 2022-11-23T03:14:27.0215289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqf0cr66q/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0215650Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0215814Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0216193Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0216371Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0216607Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwxs6zqps 2022-11-23T03:14:27.0216860Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwxs6zqps/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0217238Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0217403Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0217785Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0217965Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0218196Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp30e_my2z 2022-11-23T03:14:27.0218441Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp30e_my2z/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0218654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0218872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0219089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0219183Z ok (5.388s) 2022-11-23T03:14:27.0219189Z 2022-11-23T03:14:27.0219456Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0219557Z Ran 1 test in 5.389s 2022-11-23T03:14:27.0219562Z 2022-11-23T03:14:27.0219648Z OK 2022-11-23T03:14:27.0219653Z 2022-11-23T03:14:27.0219768Z Generating XML reports... 2022-11-23T03:14:27.0220194Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123030650.xml 2022-11-23T03:14:27.0220554Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0220717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0221101Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0221340Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0221572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnbz_20by 2022-11-23T03:14:27.0221818Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnbz_20by/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0222132Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0222137Z 2022-11-23T03:14:27.0222238Z Running tests... 2022-11-23T03:14:27.0222505Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0222804Z test_allgather_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117403 2022-11-23T03:14:27.0223011Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117404 2022-11-23T03:14:27.0223222Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 117405 2022-11-23T03:14:27.0223474Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 117406 2022-11-23T03:14:27.0223854Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0224019Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0224399Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0224576Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0224814Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph9lhxdd8 2022-11-23T03:14:27.0225067Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph9lhxdd8/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0225282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0225661Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0225833Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0226200Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0226383Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0226617Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9cfnttm4 2022-11-23T03:14:27.0226863Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9cfnttm4/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0227234Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0227403Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0227791Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0227970Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0228206Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt0d82h99 2022-11-23T03:14:27.0228452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt0d82h99/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0228821Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0228987Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0229368Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0229548Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0229784Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvt8qn2gr 2022-11-23T03:14:27.0230097Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvt8qn2gr/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0230314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0230530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0230745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0230843Z ok (8.108s) 2022-11-23T03:14:27.0230849Z 2022-11-23T03:14:27.0231120Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0231207Z Ran 1 test in 8.108s 2022-11-23T03:14:27.0231228Z 2022-11-23T03:14:27.0231299Z OK 2022-11-23T03:14:27.0231305Z 2022-11-23T03:14:27.0231422Z Generating XML reports... 2022-11-23T03:14:27.0231847Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123030700.xml 2022-11-23T03:14:27.0232266Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0232433Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0232819Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0233002Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0233238Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpazpv4sz2 2022-11-23T03:14:27.0233487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpazpv4sz2/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0233801Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0233807Z 2022-11-23T03:14:27.0233911Z Running tests... 2022-11-23T03:14:27.0234178Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0234478Z test_allreduce_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117774 2022-11-23T03:14:27.0234689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117775 2022-11-23T03:14:27.0234894Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 117776 2022-11-23T03:14:27.0235101Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 117777 2022-11-23T03:14:27.0235478Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0235643Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0236025Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0236202Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0236445Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp6m_mnf3 2022-11-23T03:14:27.0236680Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp6m_mnf3/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0236895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0237267Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0237434Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0237815Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0237993Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0238225Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_kybg3if 2022-11-23T03:14:27.0238481Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_kybg3if/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0238917Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0239085Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0239464Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0239644Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0239879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph3pvqhu1 2022-11-23T03:14:27.0240129Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph3pvqhu1/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0240502Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0240669Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0241100Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0241283Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0241519Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2zvg4i4c 2022-11-23T03:14:27.0241767Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2zvg4i4c/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0241982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0242183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0242394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0242488Z ok (5.308s) 2022-11-23T03:14:27.0242494Z 2022-11-23T03:14:27.0242773Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0242880Z Ran 1 test in 5.309s 2022-11-23T03:14:27.0242886Z 2022-11-23T03:14:27.0242970Z OK 2022-11-23T03:14:27.0242976Z 2022-11-23T03:14:27.0243092Z Generating XML reports... 2022-11-23T03:14:27.0243517Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123030712.xml 2022-11-23T03:14:27.0243892Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0244058Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0244440Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0244620Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0244855Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdspbzagp 2022-11-23T03:14:27.0245110Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdspbzagp/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0245428Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0245434Z 2022-11-23T03:14:27.0245537Z Running tests... 2022-11-23T03:14:27.0245800Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0246098Z test_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118121 2022-11-23T03:14:27.0246309Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118122 2022-11-23T03:14:27.0246515Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 118123 2022-11-23T03:14:27.0246721Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 118124 2022-11-23T03:14:27.0247092Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0247309Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0247760Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0247939Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0248179Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuicmn513 2022-11-23T03:14:27.0248427Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuicmn513/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0248641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0249017Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0249183Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0249625Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0249808Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0250042Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv_ervxvg 2022-11-23T03:14:27.0250293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv_ervxvg/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0250506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0250879Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0251049Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0251432Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0251610Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0251853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp81bb0_zp 2022-11-23T03:14:27.0252102Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp81bb0_zp/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0252318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0252689Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0252841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0253225Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0253406Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0253646Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgx6_00qz 2022-11-23T03:14:27.0253897Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgx6_00qz/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0254116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0254208Z ok (5.689s) 2022-11-23T03:14:27.0254214Z 2022-11-23T03:14:27.0254485Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0254588Z Ran 1 test in 5.689s 2022-11-23T03:14:27.0254594Z 2022-11-23T03:14:27.0254679Z OK 2022-11-23T03:14:27.0254686Z 2022-11-23T03:14:27.0254804Z Generating XML reports... 2022-11-23T03:14:27.0255232Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123030721.xml 2022-11-23T03:14:27.0255603Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0255773Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0256156Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0256402Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0256640Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjjchpgl1 2022-11-23T03:14:27.0256891Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjjchpgl1/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0257209Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0257216Z 2022-11-23T03:14:27.0257317Z Running tests... 2022-11-23T03:14:27.0257585Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0257902Z test_allreduce_basics_cuda_using_work_api (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118468 2022-11-23T03:14:27.0258097Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118469 2022-11-23T03:14:27.0258353Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 118470 2022-11-23T03:14:27.0258564Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 118471 2022-11-23T03:14:27.0258940Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0259107Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0259486Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0259666Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0259905Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjhtrinjj 2022-11-23T03:14:27.0260155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjhtrinjj/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0260373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0260747Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0260914Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0261296Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0261476Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0261713Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9xkvvvlh 2022-11-23T03:14:27.0261960Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9xkvvvlh/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0262172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0262550Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0262719Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0263104Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0263282Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0263503Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpckym54n2 2022-11-23T03:14:27.0263756Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpckym54n2/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0264132Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0264300Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0264681Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0264916Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0265152Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvdr9hrl0 2022-11-23T03:14:27.0265403Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvdr9hrl0/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0265625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0265839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0265940Z ok (6.138s) 2022-11-23T03:14:27.0265946Z 2022-11-23T03:14:27.0266219Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0266321Z Ran 1 test in 6.139s 2022-11-23T03:14:27.0266327Z 2022-11-23T03:14:27.0266417Z OK 2022-11-23T03:14:27.0266423Z 2022-11-23T03:14:27.0266538Z Generating XML reports... 2022-11-23T03:14:27.0267016Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123030731.xml 2022-11-23T03:14:27.0267398Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0267566Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0267945Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0268131Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0268371Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdnk8fq6c 2022-11-23T03:14:27.0268621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdnk8fq6c/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0268920Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0268942Z 2022-11-23T03:14:27.0269029Z Running tests... 2022-11-23T03:14:27.0269302Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0269614Z test_allreduce_basics_using_work_api (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118815 2022-11-23T03:14:27.0269840Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118816 2022-11-23T03:14:27.0270046Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 118817 2022-11-23T03:14:27.0270259Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 118818 2022-11-23T03:14:27.0270634Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0270802Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0271188Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0271379Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0271602Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeku_p5za 2022-11-23T03:14:27.0271850Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeku_p5za/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0272065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0272444Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0272611Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0272998Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0273180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0273423Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzdjb2hmh 2022-11-23T03:14:27.0273733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzdjb2hmh/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0274110Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0274282Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0274665Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0274849Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0275084Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphqrm83d8 2022-11-23T03:14:27.0275332Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphqrm83d8/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0275786Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0275965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0276361Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0276553Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0276796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvnnildd4 2022-11-23T03:14:27.0277058Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvnnildd4/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0277259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0277476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0277696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0277797Z ok (5.233s) 2022-11-23T03:14:27.0277806Z 2022-11-23T03:14:27.0278088Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0278198Z Ran 1 test in 5.233s 2022-11-23T03:14:27.0278204Z 2022-11-23T03:14:27.0278297Z OK 2022-11-23T03:14:27.0278302Z 2022-11-23T03:14:27.0278430Z Generating XML reports... 2022-11-23T03:14:27.0278865Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123030742.xml 2022-11-23T03:14:27.0279245Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0279419Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0279811Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0279996Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0280238Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqvakmuvl 2022-11-23T03:14:27.0280494Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqvakmuvl/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0280817Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0280824Z 2022-11-23T03:14:27.0280935Z Running tests... 2022-11-23T03:14:27.0281202Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0281497Z test_allreduce_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119162 2022-11-23T03:14:27.0281712Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119163 2022-11-23T03:14:27.0281921Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 119164 2022-11-23T03:14:27.0282131Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 119165 2022-11-23T03:14:27.0282494Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0282735Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0283131Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0283319Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0283562Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe108z9tg 2022-11-23T03:14:27.0283816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe108z9tg/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0284042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0284422Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0284640Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0285042Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0285223Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0285459Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx4zlc1ns 2022-11-23T03:14:27.0285713Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx4zlc1ns/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0286091Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0286266Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0286653Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0286833Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0287075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg3zuxsme 2022-11-23T03:14:27.0287332Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg3zuxsme/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0287555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0287844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0288207Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0288375Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0288758Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0288951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0289195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfuo86lcs 2022-11-23T03:14:27.0289453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfuo86lcs/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0289668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0289770Z ok (4.845s) 2022-11-23T03:14:27.0289776Z 2022-11-23T03:14:27.0290050Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0290162Z Ran 1 test in 4.845s 2022-11-23T03:14:27.0290167Z 2022-11-23T03:14:27.0290252Z OK 2022-11-23T03:14:27.0290258Z 2022-11-23T03:14:27.0290381Z Generating XML reports... 2022-11-23T03:14:27.0290809Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123030751.xml 2022-11-23T03:14:27.0291190Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0291429Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0291824Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0292005Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0292249Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm508mwwm 2022-11-23T03:14:27.0292501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm508mwwm/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0292820Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0292827Z 2022-11-23T03:14:27.0292931Z Running tests... 2022-11-23T03:14:27.0293183Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0293501Z test_allreduce_coalesced_async (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119509 2022-11-23T03:14:27.0293766Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119510 2022-11-23T03:14:27.0293989Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 119511 2022-11-23T03:14:27.0294200Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 119512 2022-11-23T03:14:27.0294585Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0294753Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0295140Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0295320Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0295560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprcz9ll0y 2022-11-23T03:14:27.0295814Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprcz9ll0y/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0296042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0296420Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0296586Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0296975Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0297155Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0297400Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprv0tidqg 2022-11-23T03:14:27.0297650Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprv0tidqg/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0297878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0298262Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0298440Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0298825Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0298991Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0299232Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyvy9v441 2022-11-23T03:14:27.0299484Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyvy9v441/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0299704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0300079Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0300313Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0300701Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0300887Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0301124Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpweg2o5qp 2022-11-23T03:14:27.0301374Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpweg2o5qp/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0301588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0301821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:14:27.0302050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T03:14:27.0302324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:14:27.0302559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T03:14:27.0302960Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T03:14:27.0303366Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T03:14:27.0303763Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T03:14:27.0304155Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T03:14:27.0304915Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T03:14:27.0305023Z warnings.warn( 2022-11-23T03:14:27.0305772Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T03:14:27.0305878Z warnings.warn( 2022-11-23T03:14:27.0306629Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T03:14:27.0306738Z warnings.warn( 2022-11-23T03:14:27.0307488Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T03:14:27.0307592Z warnings.warn( 2022-11-23T03:14:27.0307690Z ok (5.146s) 2022-11-23T03:14:27.0307697Z 2022-11-23T03:14:27.0307952Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0308057Z Ran 1 test in 5.146s 2022-11-23T03:14:27.0308063Z 2022-11-23T03:14:27.0308148Z OK 2022-11-23T03:14:27.0308154Z 2022-11-23T03:14:27.0308277Z Generating XML reports... 2022-11-23T03:14:27.0308703Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123030800.xml 2022-11-23T03:14:27.0309086Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0309259Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0309644Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0309890Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0310128Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc2z9wqaz 2022-11-23T03:14:27.0310387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc2z9wqaz/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0310704Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0310711Z 2022-11-23T03:14:27.0310813Z Running tests... 2022-11-23T03:14:27.0311090Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0311399Z test_allreduce_coalesced_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119856 2022-11-23T03:14:27.0311616Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119857 2022-11-23T03:14:27.0311876Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 119858 2022-11-23T03:14:27.0312089Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 119859 2022-11-23T03:14:27.0312472Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0312638Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0313035Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0313214Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0313437Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc218ncbw 2022-11-23T03:14:27.0313686Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc218ncbw/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0313910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0314290Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0314454Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0314841Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0315023Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0315265Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5dvkq4p1 2022-11-23T03:14:27.0315522Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5dvkq4p1/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0315738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0316124Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0316296Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0316679Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0316868Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0317108Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpix8649bt 2022-11-23T03:14:27.0317361Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpix8649bt/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0317577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0317956Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0318130Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0318521Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0318761Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0318983Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpga5x6ldh 2022-11-23T03:14:27.0319241Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpga5x6ldh/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0319454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0319548Z ok (4.932s) 2022-11-23T03:14:27.0319554Z 2022-11-23T03:14:27.0319836Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0319944Z Ran 1 test in 4.932s 2022-11-23T03:14:27.0319950Z 2022-11-23T03:14:27.0320035Z OK 2022-11-23T03:14:27.0320041Z 2022-11-23T03:14:27.0320164Z Generating XML reports... 2022-11-23T03:14:27.0320635Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123030810.xml 2022-11-23T03:14:27.0321025Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0321193Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0321580Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0321771Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0322008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnqu68x5k 2022-11-23T03:14:27.0322259Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnqu68x5k/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0322581Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0322587Z 2022-11-23T03:14:27.0322697Z Running tests... 2022-11-23T03:14:27.0322972Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0323292Z test_allreduce_coalesced_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120203 2022-11-23T03:14:27.0323502Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120204 2022-11-23T03:14:27.0323718Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 120205 2022-11-23T03:14:27.0323926Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 120206 2022-11-23T03:14:27.0324286Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0324461Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0324846Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0325038Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0325279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpez8g4p20 2022-11-23T03:14:27.0325529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpez8g4p20/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0325907Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0326081Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0326465Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0326645Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0326890Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwy7_obmw 2022-11-23T03:14:27.0327136Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwy7_obmw/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0327426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0327935Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0328106Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0328494Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0328677Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0328917Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyt59kogo 2022-11-23T03:14:27.0329177Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyt59kogo/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0329556Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0329794Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0330189Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0330355Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0330597Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprykkolxr 2022-11-23T03:14:27.0330851Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprykkolxr/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0331070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0331300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0331513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0331613Z ok (4.950s) 2022-11-23T03:14:27.0331623Z 2022-11-23T03:14:27.0331897Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0332006Z Ran 1 test in 4.950s 2022-11-23T03:14:27.0332012Z 2022-11-23T03:14:27.0332099Z OK 2022-11-23T03:14:27.0332104Z 2022-11-23T03:14:27.0332222Z Generating XML reports... 2022-11-23T03:14:27.0332654Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123030819.xml 2022-11-23T03:14:27.0333034Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0333203Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0333595Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0333781Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0334027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2jri4rbr 2022-11-23T03:14:27.0334286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2jri4rbr/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0334607Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0334613Z 2022-11-23T03:14:27.0334718Z Running tests... 2022-11-23T03:14:27.0334988Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0335292Z test_allreduce_coalesced_checks_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120550 2022-11-23T03:14:27.0335510Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120551 2022-11-23T03:14:27.0335716Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 120552 2022-11-23T03:14:27.0335923Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 120553 2022-11-23T03:14:27.0336303Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0336557Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0336944Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0337136Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0337375Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxt5h3fpm 2022-11-23T03:14:27.0337622Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxt5h3fpm/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0337846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0338224Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0338390Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0338829Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0339014Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0339259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps8nvnun9 2022-11-23T03:14:27.0339515Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps8nvnun9/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0339736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0340115Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0340285Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0340680Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0340868Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0341091Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppds_cqjb 2022-11-23T03:14:27.0341349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppds_cqjb/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0341566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0341939Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0342117Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0342502Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0342690Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0342930Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpztfsl0ci 2022-11-23T03:14:27.0343182Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpztfsl0ci/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0343405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0343502Z ok (5.130s) 2022-11-23T03:14:27.0343508Z 2022-11-23T03:14:27.0343786Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0343893Z Ran 1 test in 5.131s 2022-11-23T03:14:27.0343899Z 2022-11-23T03:14:27.0343982Z OK 2022-11-23T03:14:27.0343988Z 2022-11-23T03:14:27.0344110Z Generating XML reports... 2022-11-23T03:14:27.0344536Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123030828.xml 2022-11-23T03:14:27.0344921Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0345096Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0345549Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0345729Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0345969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu2gjmvwn 2022-11-23T03:14:27.0346205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu2gjmvwn/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0346525Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0346531Z 2022-11-23T03:14:27.0346634Z Running tests... 2022-11-23T03:14:27.0346905Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0347215Z test_allreduce_coalesced_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120897 2022-11-23T03:14:27.0347475Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120898 2022-11-23T03:14:27.0347694Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 120899 2022-11-23T03:14:27.0347903Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 120900 2022-11-23T03:14:27.0348284Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0348455Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0348847Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0349029Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0349273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1q_3z8tn 2022-11-23T03:14:27.0349527Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1q_3z8tn/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0349902Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0350077Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0350464Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0350650Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0350888Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5iwv8stt 2022-11-23T03:14:27.0351134Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5iwv8stt/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0351357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0351557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0351939Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0352114Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0352504Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0352685Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0352927Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_3gm214v 2022-11-23T03:14:27.0353174Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_3gm214v/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0353559Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0353728Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0354114Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0354360Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0354595Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnz3k0_pm 2022-11-23T03:14:27.0354848Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnz3k0_pm/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0355065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0355282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0355383Z ok (5.610s) 2022-11-23T03:14:27.0355389Z 2022-11-23T03:14:27.0355664Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0355773Z Ran 1 test in 5.611s 2022-11-23T03:14:27.0355779Z 2022-11-23T03:14:27.0355867Z OK 2022-11-23T03:14:27.0355873Z 2022-11-23T03:14:27.0356034Z Generating XML reports... 2022-11-23T03:14:27.0356473Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123030837.xml 2022-11-23T03:14:27.0356848Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0357000Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0357381Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0357570Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0357805Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb9kr4ihj 2022-11-23T03:14:27.0358061Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb9kr4ihj/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0358373Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0358386Z 2022-11-23T03:14:27.0358488Z Running tests... 2022-11-23T03:14:27.0358763Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0359058Z test_allreduce_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 121268 2022-11-23T03:14:27.0359263Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 121269 2022-11-23T03:14:27.0359479Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 121270 2022-11-23T03:14:27.0359689Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 121271 2022-11-23T03:14:27.0360063Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0360232Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0360619Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0360799Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0361032Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5gz03w47 2022-11-23T03:14:27.0361279Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5gz03w47/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0361494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0361867Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0362031Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0362398Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0362578Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0362877Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj1_5ilyx 2022-11-23T03:14:27.0363125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj1_5ilyx/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0363340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0363716Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0363883Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0364267Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0364446Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0364682Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy89nkri_ 2022-11-23T03:14:27.0364976Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy89nkri_/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0365357Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0365523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0365903Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0366084Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0366319Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxks79dxx 2022-11-23T03:14:27.0366567Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxks79dxx/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0366782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0367006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0367103Z ok (5.131s) 2022-11-23T03:14:27.0367109Z 2022-11-23T03:14:27.0367381Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0367469Z Ran 1 test in 5.132s 2022-11-23T03:14:27.0367490Z 2022-11-23T03:14:27.0367562Z OK 2022-11-23T03:14:27.0367567Z 2022-11-23T03:14:27.0367684Z Generating XML reports... 2022-11-23T03:14:27.0368179Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123030847.xml 2022-11-23T03:14:27.0368552Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0368718Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0369102Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0369281Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0369523Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1e_lz9hw 2022-11-23T03:14:27.0369770Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1e_lz9hw/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0370090Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0370096Z 2022-11-23T03:14:27.0370198Z Running tests... 2022-11-23T03:14:27.0370464Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0370764Z test_allreduce_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 121639 2022-11-23T03:14:27.0370971Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 121640 2022-11-23T03:14:27.0371176Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 121641 2022-11-23T03:14:27.0371388Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 121642 2022-11-23T03:14:27.0371837Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0372004Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0372387Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0372570Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0372808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbioiqost 2022-11-23T03:14:27.0373044Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbioiqost/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0373259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0373678Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0373851Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0374240Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0374421Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0374656Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfgf7yhur 2022-11-23T03:14:27.0374905Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfgf7yhur/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0375277Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0375442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0375823Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0376008Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0376247Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb8dkd9d3 2022-11-23T03:14:27.0376496Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb8dkd9d3/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0376716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0376934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0377308Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0377476Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0377858Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0378037Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0378274Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0_sdxvok 2022-11-23T03:14:27.0378523Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0_sdxvok/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0378724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0378819Z ok (7.841s) 2022-11-23T03:14:27.0378825Z 2022-11-23T03:14:27.0379098Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0379198Z Ran 1 test in 7.841s 2022-11-23T03:14:27.0379204Z 2022-11-23T03:14:27.0379290Z OK 2022-11-23T03:14:27.0379296Z 2022-11-23T03:14:27.0379412Z Generating XML reports... 2022-11-23T03:14:27.0379838Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123030856.xml 2022-11-23T03:14:27.0380213Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0380439Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0380827Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0381010Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0381252Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm0gx06ga 2022-11-23T03:14:27.0381501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm0gx06ga/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0381816Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0381823Z 2022-11-23T03:14:27.0381924Z Running tests... 2022-11-23T03:14:27.0382191Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0382531Z test_barrier_implies_wait (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 122010 2022-11-23T03:14:27.0382749Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 122011 2022-11-23T03:14:27.0382956Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 122012 2022-11-23T03:14:27.0383167Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 122013 2022-11-23T03:14:27.0383543Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0383696Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0384082Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0384264Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0384499Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ia7l4il 2022-11-23T03:14:27.0384753Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ia7l4il/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0384967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0385339Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0385503Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0385889Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0386069Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0386307Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps961smza 2022-11-23T03:14:27.0386554Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps961smza/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0386773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0387147Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0387315Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0387700Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0387878Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0388117Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpijli538g 2022-11-23T03:14:27.0388366Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpijli538g/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0388581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0388960Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0389187Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0389560Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0389740Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0389978Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx44c65ez 2022-11-23T03:14:27.0390229Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx44c65ez/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0390442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0390534Z ok (4.943s) 2022-11-23T03:14:27.0390540Z 2022-11-23T03:14:27.0390808Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0390909Z Ran 1 test in 4.943s 2022-11-23T03:14:27.0390919Z 2022-11-23T03:14:27.0391050Z OK 2022-11-23T03:14:27.0391057Z 2022-11-23T03:14:27.0391177Z Generating XML reports... 2022-11-23T03:14:27.0391601Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123030908.xml 2022-11-23T03:14:27.0391973Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0392142Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0392529Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0392708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0392951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbortlvnh 2022-11-23T03:14:27.0393200Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbortlvnh/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0393523Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0393530Z 2022-11-23T03:14:27.0393631Z Running tests... 2022-11-23T03:14:27.0393899Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0394191Z test_broadcast_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 122357 2022-11-23T03:14:27.0394386Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 122358 2022-11-23T03:14:27.0394592Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 122359 2022-11-23T03:14:27.0394799Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 122360 2022-11-23T03:14:27.0395171Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0395337Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0395728Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0395910Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0396147Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpriita8go 2022-11-23T03:14:27.0396397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpriita8go/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0396615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0396986Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0397151Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0397533Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0397776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0398012Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzdcx7oy3 2022-11-23T03:14:27.0398264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzdcx7oy3/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0398640Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0398809Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0399195Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0399373Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0399610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphiy7tx7h 2022-11-23T03:14:27.0399922Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphiy7tx7h/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0400128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0400342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0400719Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0400888Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0401270Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0401448Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0401683Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9abka2q6 2022-11-23T03:14:27.0401931Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9abka2q6/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0402152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0402247Z ok (5.107s) 2022-11-23T03:14:27.0402253Z 2022-11-23T03:14:27.0402526Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0402627Z Ran 1 test in 5.108s 2022-11-23T03:14:27.0402633Z 2022-11-23T03:14:27.0402720Z OK 2022-11-23T03:14:27.0402726Z 2022-11-23T03:14:27.0402843Z Generating XML reports... 2022-11-23T03:14:27.0403269Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123030917.xml 2022-11-23T03:14:27.0403640Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0403810Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0404195Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0404382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0404618Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn5uksz3b 2022-11-23T03:14:27.0404867Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn5uksz3b/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0405167Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0405189Z 2022-11-23T03:14:27.0405277Z Running tests... 2022-11-23T03:14:27.0405544Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0405842Z test_broadcast_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 122704 2022-11-23T03:14:27.0406050Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 122705 2022-11-23T03:14:27.0406256Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 122706 2022-11-23T03:14:27.0406537Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 122707 2022-11-23T03:14:27.0406913Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0407079Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0407463Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0407644Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0407942Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnjj0_2qo 2022-11-23T03:14:27.0408192Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnjj0_2qo/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0408405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0408838Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0409011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0409397Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0409575Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0409813Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6kd5ccip 2022-11-23T03:14:27.0410061Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6kd5ccip/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0410275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0410649Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0410804Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0411193Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0411373Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0411612Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk0sqzhey 2022-11-23T03:14:27.0411861Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk0sqzhey/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0412077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0412453Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0412619Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0413003Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0413185Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0413422Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx__r33_w 2022-11-23T03:14:27.0413666Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx__r33_w/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0413881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0413975Z ok (5.441s) 2022-11-23T03:14:27.0413981Z 2022-11-23T03:14:27.0414250Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0414356Z Ran 1 test in 5.441s 2022-11-23T03:14:27.0414362Z 2022-11-23T03:14:27.0414447Z OK 2022-11-23T03:14:27.0414453Z 2022-11-23T03:14:27.0414569Z Generating XML reports... 2022-11-23T03:14:27.0414998Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123030927.xml 2022-11-23T03:14:27.0415451Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0415618Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0415985Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0416163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0416400Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyrgndacf 2022-11-23T03:14:27.0416650Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyrgndacf/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0416965Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0416971Z 2022-11-23T03:14:27.0417074Z Running tests... 2022-11-23T03:14:27.0417340Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0417684Z test_broadcast_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 123051 2022-11-23T03:14:27.0417898Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 123052 2022-11-23T03:14:27.0418107Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 123053 2022-11-23T03:14:27.0418312Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 123054 2022-11-23T03:14:27.0418688Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0418856Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0419239Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0419418Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0419653Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpejikfxx8 2022-11-23T03:14:27.0419906Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpejikfxx8/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0420122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0420495Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0420665Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0421047Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0421227Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0421447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_of7ybfd 2022-11-23T03:14:27.0421695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_of7ybfd/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0421914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0422287Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0422454Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0422838Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0423014Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0423254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdf0wi8xt 2022-11-23T03:14:27.0423504Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdf0wi8xt/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0423719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0424160Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0424325Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0424709Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0424893Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0425130Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptm6afn13 2022-11-23T03:14:27.0425377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptm6afn13/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0425593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0425688Z ok (5.141s) 2022-11-23T03:14:27.0425694Z 2022-11-23T03:14:27.0425962Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0426115Z Ran 1 test in 5.141s 2022-11-23T03:14:27.0426122Z 2022-11-23T03:14:27.0426210Z OK 2022-11-23T03:14:27.0426216Z 2022-11-23T03:14:27.0426320Z Generating XML reports... 2022-11-23T03:14:27.0426746Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123030936.xml 2022-11-23T03:14:27.0427123Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0427289Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0427672Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0427852Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0428085Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmw4b_ax5 2022-11-23T03:14:27.0428335Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmw4b_ax5/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0428652Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0428658Z 2022-11-23T03:14:27.0428760Z Running tests... 2022-11-23T03:14:27.0429027Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0429317Z test_broadcast_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 123398 2022-11-23T03:14:27.0429527Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 123399 2022-11-23T03:14:27.0429736Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 123400 2022-11-23T03:14:27.0429942Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 123401 2022-11-23T03:14:27.0430315Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0430488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0430870Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0431048Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0431283Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph4ay49hy 2022-11-23T03:14:27.0431532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph4ay49hy/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0431910Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0432062Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0432442Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0432629Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0432927Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcphbg3wh 2022-11-23T03:14:27.0433177Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcphbg3wh/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0433394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0433615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0433990Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0434157Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0434545Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0434725Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0435008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt4p2ildt 2022-11-23T03:14:27.0435261Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt4p2ildt/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0435475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0435849Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0436013Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0436394Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0436575Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0436810Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_v_na_e1 2022-11-23T03:14:27.0437062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_v_na_e1/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0437279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0437360Z ok (4.987s) 2022-11-23T03:14:27.0437385Z 2022-11-23T03:14:27.0437640Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0437742Z Ran 1 test in 4.988s 2022-11-23T03:14:27.0437748Z 2022-11-23T03:14:27.0437834Z OK 2022-11-23T03:14:27.0437840Z 2022-11-23T03:14:27.0437955Z Generating XML reports... 2022-11-23T03:14:27.0438378Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123030945.xml 2022-11-23T03:14:27.0438749Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0438919Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0439304Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0439486Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0439725Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptdqat73j 2022-11-23T03:14:27.0439973Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptdqat73j/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0440288Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0440294Z 2022-11-23T03:14:27.0440396Z Running tests... 2022-11-23T03:14:27.0440663Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0440961Z test_broadcast_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 123769 2022-11-23T03:14:27.0441172Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 123770 2022-11-23T03:14:27.0441446Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 123771 2022-11-23T03:14:27.0441654Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 123772 2022-11-23T03:14:27.0442031Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0442199Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0442567Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0442746Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0442985Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu0ri9zo0 2022-11-23T03:14:27.0443232Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu0ri9zo0/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0443492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0443873Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0444038Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0444421Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0444603Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0444840Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmvh18e9p 2022-11-23T03:14:27.0445092Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmvh18e9p/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0445314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0445689Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0445858Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0446240Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0446419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0446655Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnq_d7__t 2022-11-23T03:14:27.0446899Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnq_d7__t/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0447112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0447486Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0447652Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0448172Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0448342Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0448576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw05uf_xc 2022-11-23T03:14:27.0448822Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw05uf_xc/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0449035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0449128Z ok (6.835s) 2022-11-23T03:14:27.0449134Z 2022-11-23T03:14:27.0449403Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0449503Z Ran 1 test in 6.836s 2022-11-23T03:14:27.0449509Z 2022-11-23T03:14:27.0449595Z OK 2022-11-23T03:14:27.0449601Z 2022-11-23T03:14:27.0449719Z Generating XML reports... 2022-11-23T03:14:27.0450142Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123030954.xml 2022-11-23T03:14:27.0450613Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0450781Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0451168Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0451346Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0451579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi8u_as9s 2022-11-23T03:14:27.0451826Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi8u_as9s/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0452143Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0452149Z 2022-11-23T03:14:27.0452249Z Running tests... 2022-11-23T03:14:27.0452572Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0452866Z test_empty_tensors (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 124140 2022-11-23T03:14:27.0453077Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 124141 2022-11-23T03:14:27.0453270Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 124142 2022-11-23T03:14:27.0453479Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 124143 2022-11-23T03:14:27.0453855Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0454024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0454405Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0454589Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0454828Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp38951ibl 2022-11-23T03:14:27.0455072Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp38951ibl/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0455443Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0455611Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0455992Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0456173Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0456409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsv0wlkzq 2022-11-23T03:14:27.0456659Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsv0wlkzq/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0456880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0457095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0457469Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0457634Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0458018Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0458197Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0458434Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz9pvu8se 2022-11-23T03:14:27.0458683Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz9pvu8se/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0458888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0459319Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0459490Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0459872Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0460052Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0460289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf23gxnsk 2022-11-23T03:14:27.0460536Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf23gxnsk/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0460751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0460844Z ok (4.830s) 2022-11-23T03:14:27.0460850Z 2022-11-23T03:14:27.0461165Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0461269Z Ran 1 test in 4.831s 2022-11-23T03:14:27.0461275Z 2022-11-23T03:14:27.0461360Z OK 2022-11-23T03:14:27.0461366Z 2022-11-23T03:14:27.0461483Z Generating XML reports... 2022-11-23T03:14:27.0461911Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123031005.xml 2022-11-23T03:14:27.0462283Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0462450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0462831Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0463008Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0463246Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7x2p4ntd 2022-11-23T03:14:27.0463501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7x2p4ntd/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0463816Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0463822Z 2022-11-23T03:14:27.0463925Z Running tests... 2022-11-23T03:14:27.0464177Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0464468Z test_gather_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 124487 2022-11-23T03:14:27.0464676Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 124488 2022-11-23T03:14:27.0464887Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 124489 2022-11-23T03:14:27.0465093Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 124490 2022-11-23T03:14:27.0465469Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0465639Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0466018Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0466199Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0466434Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnmmj4k02 2022-11-23T03:14:27.0466680Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnmmj4k02/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0467051Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0467215Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0467596Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0467850Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0468088Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpscgxu25y 2022-11-23T03:14:27.0468339Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpscgxu25y/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0468553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0468766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0469143Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0469309Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0469677Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0469903Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0470145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo__jssp3 2022-11-23T03:14:27.0470394Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo__jssp3/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0470767Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0470934Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0471315Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0471498Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0471735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxy53yakk 2022-11-23T03:14:27.0471986Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxy53yakk/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0472202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0472417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0472511Z ok (5.127s) 2022-11-23T03:14:27.0472517Z 2022-11-23T03:14:27.0472788Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0472887Z Ran 1 test in 5.127s 2022-11-23T03:14:27.0472893Z 2022-11-23T03:14:27.0472978Z OK 2022-11-23T03:14:27.0472984Z 2022-11-23T03:14:27.0473104Z Generating XML reports... 2022-11-23T03:14:27.0473529Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123031014.xml 2022-11-23T03:14:27.0473901Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0474068Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0474457Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0474621Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0474859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi1jdwjqo 2022-11-23T03:14:27.0475109Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi1jdwjqo/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0475421Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0475429Z 2022-11-23T03:14:27.0475531Z Running tests... 2022-11-23T03:14:27.0475798Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0476094Z test_gather_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 124834 2022-11-23T03:14:27.0476307Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 124835 2022-11-23T03:14:27.0476581Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 124836 2022-11-23T03:14:27.0476789Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 124837 2022-11-23T03:14:27.0477164Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0477331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0477712Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0477892Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0478129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4lvrpfjj 2022-11-23T03:14:27.0478378Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4lvrpfjj/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0478642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0479017Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0479191Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0479579Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0479759Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0479994Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp820d0sd3 2022-11-23T03:14:27.0480228Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp820d0sd3/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0480440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0480818Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0480987Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0481373Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0481553Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0481787Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpedamu0fk 2022-11-23T03:14:27.0482037Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpedamu0fk/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0482254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0482626Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0482792Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0483183Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0483361Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0483599Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmnrps1tn 2022-11-23T03:14:27.0483848Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmnrps1tn/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0484064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0484160Z ok (5.444s) 2022-11-23T03:14:27.0484166Z 2022-11-23T03:14:27.0484435Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0484536Z Ran 1 test in 5.444s 2022-11-23T03:14:27.0484543Z 2022-11-23T03:14:27.0484629Z OK 2022-11-23T03:14:27.0484635Z 2022-11-23T03:14:27.0484754Z Generating XML reports... 2022-11-23T03:14:27.0485233Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123031023.xml 2022-11-23T03:14:27.0485604Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0485772Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0486156Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0486336Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0486569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6l45m96l 2022-11-23T03:14:27.0486814Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6l45m96l/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0487132Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0487141Z 2022-11-23T03:14:27.0487286Z Running tests... 2022-11-23T03:14:27.0487560Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0487924Z test_gather_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 125181 2022-11-23T03:14:27.0488135Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 125182 2022-11-23T03:14:27.0488347Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 125183 2022-11-23T03:14:27.0488552Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 125184 2022-11-23T03:14:27.0488925Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0489092Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0489475Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0489665Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0489901Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsqnjgfx9 2022-11-23T03:14:27.0490151Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsqnjgfx9/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0490367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0490742Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0490894Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0491278Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0491457Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0491694Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn32_im19 2022-11-23T03:14:27.0491940Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn32_im19/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0492314Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0492480Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0492863Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0493043Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0493284Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl4rf8bw7 2022-11-23T03:14:27.0493528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl4rf8bw7/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0493747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0494037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0494414Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0494583Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0494966Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0495147Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0495383Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3z21ehp9 2022-11-23T03:14:27.0495630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3z21ehp9/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0495844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0495992Z ok (5.052s) 2022-11-23T03:14:27.0495999Z 2022-11-23T03:14:27.0496259Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0496364Z Ran 1 test in 5.053s 2022-11-23T03:14:27.0496370Z 2022-11-23T03:14:27.0496456Z OK 2022-11-23T03:14:27.0496462Z 2022-11-23T03:14:27.0496580Z Generating XML reports... 2022-11-23T03:14:27.0497008Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123031033.xml 2022-11-23T03:14:27.0497381Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0497546Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0497928Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0498107Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0498349Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa_2ki05m 2022-11-23T03:14:27.0498594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa_2ki05m/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0498910Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0498916Z 2022-11-23T03:14:27.0499016Z Running tests... 2022-11-23T03:14:27.0499286Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0499599Z test_gather_noncontiguous_input (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 125528 2022-11-23T03:14:27.0499864Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 125529 2022-11-23T03:14:27.0500095Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 125530 2022-11-23T03:14:27.0500318Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 125531 2022-11-23T03:14:27.0500715Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0500906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0501304Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0501496Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0501719Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ic78e3z 2022-11-23T03:14:27.0501983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ic78e3z/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0502220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0502613Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0502857Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0503256Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0503453Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0503707Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiko_st28 2022-11-23T03:14:27.0503967Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiko_st28/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0504193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0504582Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0504765Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0505213Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0505414Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0505660Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3u21es1k 2022-11-23T03:14:27.0505929Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3u21es1k/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0506156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0506547Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0506729Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0507127Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0507324Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0507553Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn7682acy 2022-11-23T03:14:27.0507812Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn7682acy/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0508042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0508154Z ok (4.733s) 2022-11-23T03:14:27.0508160Z 2022-11-23T03:14:27.0508448Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0508565Z Ran 1 test in 4.733s 2022-11-23T03:14:27.0508571Z 2022-11-23T03:14:27.0508675Z OK 2022-11-23T03:14:27.0508681Z 2022-11-23T03:14:27.0508810Z Generating XML reports... 2022-11-23T03:14:27.0509252Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123031043.xml 2022-11-23T03:14:27.0509641Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0509824Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0510220Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0510411Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0510658Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4hvkga8q 2022-11-23T03:14:27.0510923Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4hvkga8q/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0511250Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0511256Z 2022-11-23T03:14:27.0511373Z Running tests... 2022-11-23T03:14:27.0511661Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0511966Z test_gather_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 125875 2022-11-23T03:14:27.0512256Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 125876 2022-11-23T03:14:27.0512481Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 125877 2022-11-23T03:14:27.0512675Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 125878 2022-11-23T03:14:27.0513067Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0513250Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0513649Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0513842Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0514095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdni_15zi 2022-11-23T03:14:27.0514404Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdni_15zi/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0514645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0515037Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0515217Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0515616Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0515813Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0516061Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3b83f1hs 2022-11-23T03:14:27.0516321Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3b83f1hs/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0516553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0516949Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0517128Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0517527Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0517720Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0517966Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7otguhq2 2022-11-23T03:14:27.0518225Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7otguhq2/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0518455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0518816Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0518999Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0519398Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0519591Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0519842Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfiv3rzrj 2022-11-23T03:14:27.0520109Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfiv3rzrj/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0520337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0520445Z ok (5.330s) 2022-11-23T03:14:27.0520451Z 2022-11-23T03:14:27.0520731Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0520851Z Ran 1 test in 5.331s 2022-11-23T03:14:27.0520858Z 2022-11-23T03:14:27.0520958Z OK 2022-11-23T03:14:27.0521025Z 2022-11-23T03:14:27.0521163Z Generating XML reports... 2022-11-23T03:14:27.0521611Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123031052.xml 2022-11-23T03:14:27.0521997Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0522178Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0522574Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0522781Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0523045Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptws9hb_t 2022-11-23T03:14:27.0523313Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptws9hb_t/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0523692Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0523703Z 2022-11-23T03:14:27.0523823Z Running tests... 2022-11-23T03:14:27.0524082Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0524392Z test_gather_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 126246 2022-11-23T03:14:27.0524618Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 126247 2022-11-23T03:14:27.0524837Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 126248 2022-11-23T03:14:27.0525057Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 126249 2022-11-23T03:14:27.0525447Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0525623Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0526028Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0526223Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0526477Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmfhxfu6m 2022-11-23T03:14:27.0526742Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmfhxfu6m/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0526969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0527356Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0527537Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0528005Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0528201Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0528458Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzptprdm5 2022-11-23T03:14:27.0528729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzptprdm5/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0528962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0529355Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0529533Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0529929Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0530092Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0530347Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpthxez8pt 2022-11-23T03:14:27.0530681Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpthxez8pt/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0530914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0531312Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0531488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0531888Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0532082Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0532330Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpitz_ejn4 2022-11-23T03:14:27.0532592Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpitz_ejn4/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0532884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0532997Z ok (9.260s) 2022-11-23T03:14:27.0533003Z 2022-11-23T03:14:27.0533290Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0533404Z Ran 1 test in 9.260s 2022-11-23T03:14:27.0533410Z 2022-11-23T03:14:27.0533501Z OK 2022-11-23T03:14:27.0533507Z 2022-11-23T03:14:27.0533628Z Generating XML reports... 2022-11-23T03:14:27.0534061Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123031101.xml 2022-11-23T03:14:27.0534440Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0534615Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0535007Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0535204Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0535427Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp11ct6qqz 2022-11-23T03:14:27.0535683Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp11ct6qqz/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0536004Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0536010Z 2022-11-23T03:14:27.0536119Z Running tests... 2022-11-23T03:14:27.0536397Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0536708Z test_multi_device_constructor (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 126617 2022-11-23T03:14:27.0536926Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 126618 2022-11-23T03:14:27.0537141Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 126619 2022-11-23T03:14:27.0537368Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 126620 2022-11-23T03:14:27.0537753Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0537926Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0538320Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0538508Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0538749Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd0ubtev3 2022-11-23T03:14:27.0539004Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd0ubtev3/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0539223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0539606Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0539846Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0540241Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0540429Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0540671Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpto5v56i4 2022-11-23T03:14:27.0540925Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpto5v56i4/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0541127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0541513Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0541688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0542130Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0542320Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0542565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqup_75ab 2022-11-23T03:14:27.0542818Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqup_75ab/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0555625Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0555821Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0556249Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0556426Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0556654Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuqja5h1x 2022-11-23T03:14:27.0556905Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuqja5h1x/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0557116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0557326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0557417Z ok (5.034s) 2022-11-23T03:14:27.0557424Z 2022-11-23T03:14:27.0557692Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0557789Z Ran 1 test in 5.035s 2022-11-23T03:14:27.0557795Z 2022-11-23T03:14:27.0557875Z OK 2022-11-23T03:14:27.0557881Z 2022-11-23T03:14:27.0557992Z Generating XML reports... 2022-11-23T03:14:27.0558412Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123031115.xml 2022-11-23T03:14:27.0558782Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0558944Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0559324Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0559498Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0559729Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprw5ka_z0 2022-11-23T03:14:27.0559972Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprw5ka_z0/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0560280Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0560286Z 2022-11-23T03:14:27.0560384Z Running tests... 2022-11-23T03:14:27.0560647Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0560930Z test_reduce_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 126968 2022-11-23T03:14:27.0561246Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 126969 2022-11-23T03:14:27.0561437Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 126970 2022-11-23T03:14:27.0561640Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 126971 2022-11-23T03:14:27.0562014Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0562174Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0562552Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0562724Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0563002Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaucnvxx1 2022-11-23T03:14:27.0563254Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaucnvxx1/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0563624Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0563785Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0564159Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0564330Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0564559Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9afyc4ol 2022-11-23T03:14:27.0564803Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9afyc4ol/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0565178Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0565342Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0565724Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0565898Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0566130Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3xdxc6_z 2022-11-23T03:14:27.0566373Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3xdxc6_z/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0566582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0566792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0567149Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0567314Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0567873Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0568052Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0568283Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmikv4q32 2022-11-23T03:14:27.0568525Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmikv4q32/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0568731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0568941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0569029Z ok (4.833s) 2022-11-23T03:14:27.0569035Z 2022-11-23T03:14:27.0569302Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0569399Z Ran 1 test in 4.834s 2022-11-23T03:14:27.0569479Z 2022-11-23T03:14:27.0569560Z OK 2022-11-23T03:14:27.0569566Z 2022-11-23T03:14:27.0569676Z Generating XML reports... 2022-11-23T03:14:27.0570100Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123031124.xml 2022-11-23T03:14:27.0570465Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0570626Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0571002Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0571174Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0571404Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoh09zplm 2022-11-23T03:14:27.0571646Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoh09zplm/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0572011Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0572019Z 2022-11-23T03:14:27.0572117Z Running tests... 2022-11-23T03:14:27.0572373Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0572665Z test_reduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 127315 2022-11-23T03:14:27.0572869Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 127316 2022-11-23T03:14:27.0573073Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 127317 2022-11-23T03:14:27.0573276Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 127318 2022-11-23T03:14:27.0573644Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0573808Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0574186Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0574360Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0574592Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzyxfg1dc 2022-11-23T03:14:27.0574840Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzyxfg1dc/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0575046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0575414Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0575575Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0575950Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0576126Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0576358Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo10qvjve 2022-11-23T03:14:27.0576602Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo10qvjve/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0576967Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0577126Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0577503Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0577667Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0577901Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi3qiei9d 2022-11-23T03:14:27.0578147Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi3qiei9d/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0578413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0578784Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0578943Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0579314Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0579482Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0579710Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkus3puus 2022-11-23T03:14:27.0579951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkus3puus/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0580199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0580415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0580499Z ok (6.736s) 2022-11-23T03:14:27.0580505Z 2022-11-23T03:14:27.0580767Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0580865Z Ran 1 test in 6.737s 2022-11-23T03:14:27.0580871Z 2022-11-23T03:14:27.0580950Z OK 2022-11-23T03:14:27.0580956Z 2022-11-23T03:14:27.0581065Z Generating XML reports... 2022-11-23T03:14:27.0581482Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123031133.xml 2022-11-23T03:14:27.0581849Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0582007Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0582386Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0582554Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0582783Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2rhxs2_q 2022-11-23T03:14:27.0583026Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2rhxs2_q/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0583329Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0583335Z 2022-11-23T03:14:27.0583432Z Running tests... 2022-11-23T03:14:27.0583691Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0583976Z test_reduce_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 127662 2022-11-23T03:14:27.0584182Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 127663 2022-11-23T03:14:27.0584385Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 127664 2022-11-23T03:14:27.0584589Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 127665 2022-11-23T03:14:27.0584954Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0585112Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0585487Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0585658Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0585887Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp00gg2pq6 2022-11-23T03:14:27.0586130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp00gg2pq6/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0586339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0586771Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0586932Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0587308Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0587482Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0587712Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp807m__aq 2022-11-23T03:14:27.0587943Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp807m__aq/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0588151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0588517Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0588723Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0589106Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0589278Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0589507Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbmf8z8ki 2022-11-23T03:14:27.0589749Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbmf8z8ki/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0589957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0590324Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0590483Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0590862Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0591038Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0591268Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsft1c9ok 2022-11-23T03:14:27.0591509Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsft1c9ok/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0591720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0591810Z ok (4.946s) 2022-11-23T03:14:27.0591816Z 2022-11-23T03:14:27.0592081Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0592179Z Ran 1 test in 4.946s 2022-11-23T03:14:27.0592185Z 2022-11-23T03:14:27.0592265Z OK 2022-11-23T03:14:27.0592270Z 2022-11-23T03:14:27.0592381Z Generating XML reports... 2022-11-23T03:14:27.0592789Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123031144.xml 2022-11-23T03:14:27.0593160Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0593321Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0593697Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0593870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0594100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaypk3tkg 2022-11-23T03:14:27.0594343Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaypk3tkg/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0594647Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0594653Z 2022-11-23T03:14:27.0594747Z Running tests... 2022-11-23T03:14:27.0595008Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0595344Z test_reduce_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 128009 2022-11-23T03:14:27.0595546Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 128010 2022-11-23T03:14:27.0595749Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 128011 2022-11-23T03:14:27.0595949Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 128012 2022-11-23T03:14:27.0596319Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0596480Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0596855Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0597026Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0597302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8hf5zvu0 2022-11-23T03:14:27.0597550Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8hf5zvu0/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0597762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0598130Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0598281Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0598658Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0598831Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0599059Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpubb5nfz9 2022-11-23T03:14:27.0599304Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpubb5nfz9/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0599516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0599882Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0600041Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0600416Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0600588Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0600816Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwthxnylg 2022-11-23T03:14:27.0601057Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwthxnylg/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0601269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0601637Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0601797Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0602171Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0602342Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0602575Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph5btyak9 2022-11-23T03:14:27.0602817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph5btyak9/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0603024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0603113Z ok (5.454s) 2022-11-23T03:14:27.0603119Z 2022-11-23T03:14:27.0603376Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0603533Z Ran 1 test in 5.455s 2022-11-23T03:14:27.0603540Z 2022-11-23T03:14:27.0603621Z OK 2022-11-23T03:14:27.0603626Z 2022-11-23T03:14:27.0603740Z Generating XML reports... 2022-11-23T03:14:27.0604166Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123031153.xml 2022-11-23T03:14:27.0604535Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0604697Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0605075Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0605250Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0605484Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzu5273hx 2022-11-23T03:14:27.0605777Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzu5273hx/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0606089Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0606095Z 2022-11-23T03:14:27.0606194Z Running tests... 2022-11-23T03:14:27.0606458Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0606751Z test_reduce_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 128380 2022-11-23T03:14:27.0606956Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 128381 2022-11-23T03:14:27.0607157Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 128382 2022-11-23T03:14:27.0607361Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 128383 2022-11-23T03:14:27.0607799Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0607970Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0608352Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0608530Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0608752Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj7kx5dws 2022-11-23T03:14:27.0609000Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj7kx5dws/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0609212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0609583Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0609747Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0610128Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0610306Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0610537Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn3wp8jaz 2022-11-23T03:14:27.0610786Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn3wp8jaz/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0611154Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0611322Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0611699Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0611875Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0612109Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsepk8od8 2022-11-23T03:14:27.0612443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsepk8od8/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0612657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0612869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0613245Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0613410Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0613790Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0613968Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0614190Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfrh3_97c 2022-11-23T03:14:27.0614485Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfrh3_97c/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0614703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0614795Z ok (11.761s) 2022-11-23T03:14:27.0614802Z 2022-11-23T03:14:27.0615075Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0615181Z Ran 1 test in 11.761s 2022-11-23T03:14:27.0615187Z 2022-11-23T03:14:27.0615271Z OK 2022-11-23T03:14:27.0615277Z 2022-11-23T03:14:27.0615392Z Generating XML reports... 2022-11-23T03:14:27.0615812Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123031203.xml 2022-11-23T03:14:27.0616183Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0616348Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0616737Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0616914Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0617145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvlwpvfnf 2022-11-23T03:14:27.0617396Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvlwpvfnf/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0617708Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0617714Z 2022-11-23T03:14:27.0617816Z Running tests... 2022-11-23T03:14:27.0618086Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0618366Z test_round_robin (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 128751 2022-11-23T03:14:27.0618574Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 128752 2022-11-23T03:14:27.0618783Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 128753 2022-11-23T03:14:27.0618975Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 128754 2022-11-23T03:14:27.0619346Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0619512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0619894Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0620073Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0620307Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp15uilxau 2022-11-23T03:14:27.0620554Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp15uilxau/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0620772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0621208Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0621370Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0621751Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0621927Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0622166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphzv7de58 2022-11-23T03:14:27.0622413Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphzv7de58/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0622626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0623038Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0623207Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0623594Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0623772Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0624005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbhqj8yvk 2022-11-23T03:14:27.0624253Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbhqj8yvk/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0624622Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0624774Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0625158Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0625345Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0625579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfvikptvw 2022-11-23T03:14:27.0625825Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfvikptvw/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0626040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0626252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0626784Z [W ProcessGroupRoundRobin.cpp:10] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin) 2022-11-23T03:14:27.0627298Z [W ProcessGroupRoundRobin.cpp:10] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin) 2022-11-23T03:14:27.0627828Z [W ProcessGroupRoundRobin.cpp:10] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin) 2022-11-23T03:14:27.0628346Z [W ProcessGroupRoundRobin.cpp:10] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin) 2022-11-23T03:14:27.0628440Z ok (5.040s) 2022-11-23T03:14:27.0628446Z 2022-11-23T03:14:27.0628721Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0628876Z Ran 1 test in 5.040s 2022-11-23T03:14:27.0628882Z 2022-11-23T03:14:27.0628968Z OK 2022-11-23T03:14:27.0628974Z 2022-11-23T03:14:27.0629088Z Generating XML reports... 2022-11-23T03:14:27.0629515Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123031219.xml 2022-11-23T03:14:27.0629884Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0630048Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0630429Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0630611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0630845Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqgobeqf0 2022-11-23T03:14:27.0631165Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqgobeqf0/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0631486Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0631492Z 2022-11-23T03:14:27.0631594Z Running tests... 2022-11-23T03:14:27.0631856Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0632163Z test_round_robin_create_destroy (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 129110 2022-11-23T03:14:27.0632373Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 129111 2022-11-23T03:14:27.0632577Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 129112 2022-11-23T03:14:27.0632783Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 129113 2022-11-23T03:14:27.0633143Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0633312Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0633692Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0633872Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0634100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8gnu8ikz 2022-11-23T03:14:27.0634343Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8gnu8ikz/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0634556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0634926Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0635087Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0635469Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0635651Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0635881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj42x_uwk 2022-11-23T03:14:27.0636123Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj42x_uwk/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0636336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0636707Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0636872Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0637250Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0637431Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0637721Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8zzl5yvm 2022-11-23T03:14:27.0637967Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8zzl5yvm/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0638340Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0638504Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0638869Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0639045Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0639278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptry5wte_ 2022-11-23T03:14:27.0639525Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptry5wte_/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0639786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0639999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0640516Z [W ProcessGroupRoundRobin.cpp:10] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin) 2022-11-23T03:14:27.0641024Z [W ProcessGroupRoundRobin.cpp:10] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin) 2022-11-23T03:14:27.0641537Z [W ProcessGroupRoundRobin.cpp:10] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin) 2022-11-23T03:14:27.0642048Z [W ProcessGroupRoundRobin.cpp:10] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin) 2022-11-23T03:14:27.0642553Z [W ProcessGroupRoundRobin.cpp:10] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin) 2022-11-23T03:14:27.0643077Z [W ProcessGroupRoundRobin.cpp:10] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin) 2022-11-23T03:14:27.0643588Z [W ProcessGroupRoundRobin.cpp:10] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin) 2022-11-23T03:14:27.0644096Z [W ProcessGroupRoundRobin.cpp:10] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin) 2022-11-23T03:14:27.0644190Z ok (5.132s) 2022-11-23T03:14:27.0644196Z 2022-11-23T03:14:27.0644472Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0644621Z Ran 1 test in 5.132s 2022-11-23T03:14:27.0644627Z 2022-11-23T03:14:27.0644699Z OK 2022-11-23T03:14:27.0644718Z 2022-11-23T03:14:27.0644822Z Generating XML reports... 2022-11-23T03:14:27.0645251Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123031228.xml 2022-11-23T03:14:27.0645620Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0645784Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0646164Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0646343Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0646577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp97i22slc 2022-11-23T03:14:27.0646869Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp97i22slc/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0647191Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0647197Z 2022-11-23T03:14:27.0647296Z Running tests... 2022-11-23T03:14:27.0647564Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0647907Z test_scatter_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 129493 2022-11-23T03:14:27.0648114Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 129494 2022-11-23T03:14:27.0648316Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 129495 2022-11-23T03:14:27.0648520Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 129496 2022-11-23T03:14:27.0648895Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0649067Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0649445Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0649619Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0649855Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxpk5j69p 2022-11-23T03:14:27.0650104Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxpk5j69p/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0650306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0650676Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0650838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0651218Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0651398Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0651632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz7c79aav 2022-11-23T03:14:27.0651877Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz7c79aav/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0652089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0652459Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0652626Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0653006Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0653186Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0653482Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3puo9_un 2022-11-23T03:14:27.0653728Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3puo9_un/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0653941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0654314Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0654478Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0654916Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0655126Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0655403Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp16nr6fvc 2022-11-23T03:14:27.0655760Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp16nr6fvc/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0656008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0656117Z ok (5.171s) 2022-11-23T03:14:27.0656124Z 2022-11-23T03:14:27.0656448Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0656572Z Ran 1 test in 5.171s 2022-11-23T03:14:27.0656580Z 2022-11-23T03:14:27.0656674Z OK 2022-11-23T03:14:27.0656681Z 2022-11-23T03:14:27.0656818Z Generating XML reports... 2022-11-23T03:14:27.0657324Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123031237.xml 2022-11-23T03:14:27.0657769Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0657970Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0658422Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0658635Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0658909Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi3qusjfl 2022-11-23T03:14:27.0659201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi3qusjfl/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0659569Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0659577Z 2022-11-23T03:14:27.0659697Z Running tests... 2022-11-23T03:14:27.0660012Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0660363Z test_scatter_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 129840 2022-11-23T03:14:27.0660607Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 129841 2022-11-23T03:14:27.0660858Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 129842 2022-11-23T03:14:27.0661104Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 129843 2022-11-23T03:14:27.0661547Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0661745Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0662191Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0662405Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0662681Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0i_txhm6 2022-11-23T03:14:27.0662968Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0i_txhm6/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0663415Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0663675Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0664135Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0664348Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0664626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp6bwu1pd 2022-11-23T03:14:27.0664927Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp6bwu1pd/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0665177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0665433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0665876Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0666127Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0666588Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0666798Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0667076Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg8_aw086 2022-11-23T03:14:27.0667368Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg8_aw086/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0667813Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0668007Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0668464Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0668693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0668959Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphefqn2tb 2022-11-23T03:14:27.0669249Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphefqn2tb/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0669503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0669752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0669859Z ok (5.689s) 2022-11-23T03:14:27.0669866Z 2022-11-23T03:14:27.0670183Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0670309Z Ran 1 test in 5.690s 2022-11-23T03:14:27.0670316Z 2022-11-23T03:14:27.0670414Z OK 2022-11-23T03:14:27.0670420Z 2022-11-23T03:14:27.0670554Z Generating XML reports... 2022-11-23T03:14:27.0671063Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123031246.xml 2022-11-23T03:14:27.0671506Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0671706Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0672161Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0672374Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0672649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9dwc_im1 2022-11-23T03:14:27.0672938Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9dwc_im1/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0673307Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0673314Z 2022-11-23T03:14:27.0673435Z Running tests... 2022-11-23T03:14:27.0673758Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0674173Z test_scatter_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 130187 2022-11-23T03:14:27.0674415Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 130188 2022-11-23T03:14:27.0674642Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 130189 2022-11-23T03:14:27.0674880Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 130190 2022-11-23T03:14:27.0675329Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0675530Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0675986Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0676200Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0676551Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp48sckbeu 2022-11-23T03:14:27.0676840Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp48sckbeu/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0677289Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0677451Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0677827Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0678002Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0678231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqxfpoz38 2022-11-23T03:14:27.0678474Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqxfpoz38/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0678691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0678904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0679272Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0679434Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0679811Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0679985Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0680216Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9flu752b 2022-11-23T03:14:27.0680450Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9flu752b/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0680821Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0680986Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0681363Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0681537Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0681768Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz7toqqj0 2022-11-23T03:14:27.0682011Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz7toqqj0/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0682222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0682438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0682532Z ok (4.992s) 2022-11-23T03:14:27.0682538Z 2022-11-23T03:14:27.0682866Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0682966Z Ran 1 test in 4.992s 2022-11-23T03:14:27.0682972Z 2022-11-23T03:14:27.0683055Z OK 2022-11-23T03:14:27.0683061Z 2022-11-23T03:14:27.0683177Z Generating XML reports... 2022-11-23T03:14:27.0683600Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123031256.xml 2022-11-23T03:14:27.0683969Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0684132Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0684511Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0684687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0684921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm_5siylk 2022-11-23T03:14:27.0685214Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm_5siylk/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0685536Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0685543Z 2022-11-23T03:14:27.0685631Z Running tests... 2022-11-23T03:14:27.0685898Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0686188Z test_scatter_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 130534 2022-11-23T03:14:27.0686395Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 130535 2022-11-23T03:14:27.0686600Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 130536 2022-11-23T03:14:27.0686808Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 130537 2022-11-23T03:14:27.0687183Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0687348Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0687858Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0688039Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0688273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphwrkjxlo 2022-11-23T03:14:27.0688520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphwrkjxlo/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0688731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0689106Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0689270Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0689662Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0689841Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0690073Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptvo5w35w 2022-11-23T03:14:27.0690316Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptvo5w35w/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0690528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0690897Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0691049Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0691431Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0691611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0691913Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3xrg_1lt 2022-11-23T03:14:27.0692155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3xrg_1lt/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0692529Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0692692Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0693072Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0693247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0693481Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp630z85c2 2022-11-23T03:14:27.0693725Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp630z85c2/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0694136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0694355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0694446Z ok (5.628s) 2022-11-23T03:14:27.0694453Z 2022-11-23T03:14:27.0694725Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0694824Z Ran 1 test in 5.628s 2022-11-23T03:14:27.0694830Z 2022-11-23T03:14:27.0694915Z OK 2022-11-23T03:14:27.0694920Z 2022-11-23T03:14:27.0695036Z Generating XML reports... 2022-11-23T03:14:27.0695457Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123031305.xml 2022-11-23T03:14:27.0695826Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0695988Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0696364Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0696541Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0696770Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt_jo20k_ 2022-11-23T03:14:27.0697014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt_jo20k_/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0697325Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0697332Z 2022-11-23T03:14:27.0697431Z Running tests... 2022-11-23T03:14:27.0697695Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0697986Z test_scatter_stress_cuda (__main__.ProcessGroupGlooTest) ... skip: Test is flaky, see https://github.com/pytorch/pytorch/issues/15963 (0.001s) 2022-11-23T03:14:27.0697992Z 2022-11-23T03:14:27.0698262Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0698363Z Ran 1 test in 0.001s 2022-11-23T03:14:27.0698368Z 2022-11-23T03:14:27.0698465Z OK (skipped=1) 2022-11-23T03:14:27.0698471Z 2022-11-23T03:14:27.0698588Z Generating XML reports... 2022-11-23T03:14:27.0699009Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123031315.xml 2022-11-23T03:14:27.0699381Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0699545Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0699925Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0700100Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0700330Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdl_o9x59 2022-11-23T03:14:27.0700633Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdl_o9x59/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0700948Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0700954Z 2022-11-23T03:14:27.0701053Z Running tests... 2022-11-23T03:14:27.0701321Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0701609Z test_send_recv_all_to_all (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 130971 2022-11-23T03:14:27.0701803Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 130972 2022-11-23T03:14:27.0702011Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 130973 2022-11-23T03:14:27.0702212Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 130974 2022-11-23T03:14:27.0702629Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0702798Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0703182Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0703358Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0703586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp08_tdhzr 2022-11-23T03:14:27.0703834Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp08_tdhzr/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0704207Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0704372Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0704762Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0704944Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0705178Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbp0cj49d 2022-11-23T03:14:27.0705421Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbp0cj49d/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0705633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0705843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0706215Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0706380Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0706762Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0706944Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0707180Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqboi7_e6 2022-11-23T03:14:27.0707412Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqboi7_e6/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0707783Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0707948Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0708329Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0708504Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0708738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz4pv338v 2022-11-23T03:14:27.0708983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz4pv338v/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0709258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0709470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0709563Z ok (5.510s) 2022-11-23T03:14:27.0709569Z 2022-11-23T03:14:27.0709843Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0709942Z Ran 1 test in 5.510s 2022-11-23T03:14:27.0709948Z 2022-11-23T03:14:27.0710035Z OK 2022-11-23T03:14:27.0710040Z 2022-11-23T03:14:27.0710156Z Generating XML reports... 2022-11-23T03:14:27.0710580Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123031319.xml 2022-11-23T03:14:27.0710953Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0711116Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0711544Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0711722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0711951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpegv4dcn9 2022-11-23T03:14:27.0712202Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpegv4dcn9/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0712502Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0712525Z 2022-11-23T03:14:27.0712612Z Running tests... 2022-11-23T03:14:27.0712877Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0713134Z test_sparse_allreduce_basics (__main__.ProcessGroupGlooTest) ... skip: intermittent failures on Windows, in CI (0.000s) 2022-11-23T03:14:27.0713140Z 2022-11-23T03:14:27.0713408Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0713511Z Ran 1 test in 0.001s 2022-11-23T03:14:27.0713517Z 2022-11-23T03:14:27.0713615Z OK (skipped=1) 2022-11-23T03:14:27.0713620Z 2022-11-23T03:14:27.0713734Z Generating XML reports... 2022-11-23T03:14:27.0714154Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123031328.xml 2022-11-23T03:14:27.0714525Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0714688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0715069Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0715245Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0715479Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcrk1cove 2022-11-23T03:14:27.0715730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcrk1cove/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0716042Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0716048Z 2022-11-23T03:14:27.0716147Z Running tests... 2022-11-23T03:14:27.0716411Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0716717Z test_sparse_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 612 2022-11-23T03:14:27.0716922Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 613 2022-11-23T03:14:27.0717123Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 614 2022-11-23T03:14:27.0717325Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 615 2022-11-23T03:14:27.0717688Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0717905Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0718291Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0718469Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0718702Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplihl1r91 2022-11-23T03:14:27.0718948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplihl1r91/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0719162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0719532Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0719698Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0720125Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0720303Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0720538Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1nmry5i7 2022-11-23T03:14:27.0720784Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1nmry5i7/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0720993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0721367Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0721532Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0721914Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0722101Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0722333Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi810x92a 2022-11-23T03:14:27.0722576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi810x92a/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0722944Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0723105Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0723472Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0723646Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0723879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp19lunq12 2022-11-23T03:14:27.0724124Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp19lunq12/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0724338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0724546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0724637Z ok (6.091s) 2022-11-23T03:14:27.0724643Z 2022-11-23T03:14:27.0724912Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0725008Z Ran 1 test in 6.092s 2022-11-23T03:14:27.0725014Z 2022-11-23T03:14:27.0725095Z OK 2022-11-23T03:14:27.0725101Z 2022-11-23T03:14:27.0725214Z Generating XML reports... 2022-11-23T03:14:27.0725635Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123031332.xml 2022-11-23T03:14:27.0726004Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0726164Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0726599Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0726773Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0727007Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprl7dmx35 2022-11-23T03:14:27.0727252Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprl7dmx35/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0727562Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0727568Z 2022-11-23T03:14:27.0727668Z Running tests... 2022-11-23T03:14:27.0728007Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0728298Z test_sparse_allreduce_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1342 2022-11-23T03:14:27.0728557Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1343 2022-11-23T03:14:27.0728772Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1344 2022-11-23T03:14:27.0728975Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1345 2022-11-23T03:14:27.0729349Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0729512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0729893Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0730072Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0730310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg8af6zou 2022-11-23T03:14:27.0730557Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg8af6zou/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0730777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:14:27.0731146Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0731308Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0731692Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0731868Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0732101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph3svgji1 2022-11-23T03:14:27.0732349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph3svgji1/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0732560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:14:27.0732935Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0733100Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0733479Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0733654Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0733875Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2p7t8atc 2022-11-23T03:14:27.0734117Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2p7t8atc/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0734325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:14:27.0734695Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0734861Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0735305Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0735476Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0735704Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq_mgc9of 2022-11-23T03:14:27.0735942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq_mgc9of/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0736151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:14:27.0736239Z ok (5.034s) 2022-11-23T03:14:27.0736245Z 2022-11-23T03:14:27.0736508Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0736607Z Ran 1 test in 5.034s 2022-11-23T03:14:27.0736613Z 2022-11-23T03:14:27.0736692Z OK 2022-11-23T03:14:27.0736698Z 2022-11-23T03:14:27.0736809Z Generating XML reports... 2022-11-23T03:14:27.0737274Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221123031343.xml 2022-11-23T03:14:27.0737646Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0737806Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0738184Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0738360Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0738589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2kl1isr9 2022-11-23T03:14:27.0738823Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2kl1isr9/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0739130Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0739139Z 2022-11-23T03:14:27.0739238Z Running tests... 2022-11-23T03:14:27.0739501Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0739657Z test_forward_backward (__main__.ReducerTest) ... ok (0.013s) 2022-11-23T03:14:27.0739663Z 2022-11-23T03:14:27.0739925Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0740019Z Ran 1 test in 0.022s 2022-11-23T03:14:27.0740024Z 2022-11-23T03:14:27.0740103Z OK 2022-11-23T03:14:27.0740109Z 2022-11-23T03:14:27.0740220Z Generating XML reports... 2022-11-23T03:14:27.0740607Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221123031352.xml 2022-11-23T03:14:27.0740971Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0741130Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0741511Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0741687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0741919Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8pvzvyrt 2022-11-23T03:14:27.0742162Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8pvzvyrt/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0742470Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0742476Z 2022-11-23T03:14:27.0742572Z Running tests... 2022-11-23T03:14:27.0742836Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0743665Z test_forward_backward_optimizer (__main__.ReducerTest) ... [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T03:14:27.0743827Z ok (0.024s) 2022-11-23T03:14:27.0743834Z 2022-11-23T03:14:27.0744098Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0744194Z Ran 1 test in 0.032s 2022-11-23T03:14:27.0744200Z 2022-11-23T03:14:27.0744279Z OK 2022-11-23T03:14:27.0744284Z 2022-11-23T03:14:27.0744397Z Generating XML reports... 2022-11-23T03:14:27.0744782Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221123031356.xml 2022-11-23T03:14:27.0745140Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0745346Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0745735Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0745909Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0746139Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp299gfdir 2022-11-23T03:14:27.0746381Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp299gfdir/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0746688Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0746694Z 2022-11-23T03:14:27.0746790Z Running tests... 2022-11-23T03:14:27.0747051Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0747240Z test_forward_backward_unused_parameters (__main__.ReducerTest) ... ok (0.016s) 2022-11-23T03:14:27.0747245Z 2022-11-23T03:14:27.0747510Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0747612Z Ran 1 test in 0.022s 2022-11-23T03:14:27.0747618Z 2022-11-23T03:14:27.0747703Z OK 2022-11-23T03:14:27.0747709Z 2022-11-23T03:14:27.0747825Z Generating XML reports... 2022-11-23T03:14:27.0748210Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221123031400.xml 2022-11-23T03:14:27.0748579Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0748745Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0749125Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0749300Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0749536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm6jdd1g_ 2022-11-23T03:14:27.0749788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm6jdd1g_/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0750088Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0750103Z 2022-11-23T03:14:27.0750190Z Running tests... 2022-11-23T03:14:27.0750457Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0750627Z test_multi_dtype_multi_bucket (__main__.ReducerTest) ... ok (0.007s) 2022-11-23T03:14:27.0750632Z 2022-11-23T03:14:27.0750896Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0750994Z Ran 1 test in 0.022s 2022-11-23T03:14:27.0750999Z 2022-11-23T03:14:27.0751084Z OK 2022-11-23T03:14:27.0751090Z 2022-11-23T03:14:27.0751203Z Generating XML reports... 2022-11-23T03:14:27.0751589Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221123031404.xml 2022-11-23T03:14:27.0751962Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0752182Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0752571Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0752749Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0752983Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpauabsc_4 2022-11-23T03:14:27.0753232Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpauabsc_4/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0753542Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0753548Z 2022-11-23T03:14:27.0753647Z Running tests... 2022-11-23T03:14:27.0753912Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0754132Z test_multi_dtype_single_bucket (__main__.ReducerTest) ... ok (0.014s) 2022-11-23T03:14:27.0754143Z 2022-11-23T03:14:27.0754412Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0754513Z Ran 1 test in 0.022s 2022-11-23T03:14:27.0754519Z 2022-11-23T03:14:27.0754603Z OK 2022-11-23T03:14:27.0754609Z 2022-11-23T03:14:27.0754711Z Generating XML reports... 2022-11-23T03:14:27.0755101Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221123031408.xml 2022-11-23T03:14:27.0755470Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0755637Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0756021Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0756195Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0756437Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyd6oc1v0 2022-11-23T03:14:27.0756687Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyd6oc1v0/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0756996Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0757002Z 2022-11-23T03:14:27.0757101Z Running tests... 2022-11-23T03:14:27.0757368Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0757543Z test_single_dtype_single_bucket (__main__.ReducerTest) ... ok (0.006s) 2022-11-23T03:14:27.0757549Z 2022-11-23T03:14:27.0757813Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0757913Z Ran 1 test in 0.012s 2022-11-23T03:14:27.0757919Z 2022-11-23T03:14:27.0758000Z OK 2022-11-23T03:14:27.0758006Z 2022-11-23T03:14:27.0758121Z Generating XML reports... 2022-11-23T03:14:27.0758511Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221123031412.xml 2022-11-23T03:14:27.0758881Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0759045Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0759424Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0759603Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0759826Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz3c372cj 2022-11-23T03:14:27.0760067Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz3c372cj/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0760378Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0760384Z 2022-11-23T03:14:27.0760487Z Running tests... 2022-11-23T03:14:27.0760817Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0761111Z test_logging_init (__main__.RendezvousEnvTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:14:27.0761510Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:14:27.0761600Z ok (0.598s) 2022-11-23T03:14:27.0761606Z 2022-11-23T03:14:27.0761871Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0761971Z Ran 1 test in 0.599s 2022-11-23T03:14:27.0761977Z 2022-11-23T03:14:27.0762060Z OK 2022-11-23T03:14:27.0762066Z 2022-11-23T03:14:27.0762181Z Generating XML reports... 2022-11-23T03:14:27.0762587Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-RendezvousEnvTest-20221123031416.xml 2022-11-23T03:14:27.0763010Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:14:27.0763183Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:14:27.0763568Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:14:27.0763747Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:14:27.0763982Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl7m0n5xu 2022-11-23T03:14:27.0764229Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl7m0n5xu/_remote_module_non_scriptable.py 2022-11-23T03:14:27.0764542Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-11-23T03:14:27.0764548Z 2022-11-23T03:14:27.0764649Z Running tests... 2022-11-23T03:14:27.0764915Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0765801Z test_default_store_timeout_gloo (__main__.TimeoutTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/74714 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.645s) 2022-11-23T03:14:27.0765812Z 2022-11-23T03:14:27.0766078Z ---------------------------------------------------------------------- 2022-11-23T03:14:27.0766179Z Ran 1 test in 0.645s 2022-11-23T03:14:27.0766185Z 2022-11-23T03:14:27.0766280Z OK (skipped=1) 2022-11-23T03:14:27.0766286Z 2022-11-23T03:14:27.0766389Z Generating XML reports... 2022-11-23T03:14:27.0766775Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-TimeoutTest-20221123031421.xml 2022-11-23T03:14:27.0766781Z 2022-11-23T03:14:27.0767193Z ##[endgroup] 2022-11-23T03:14:27.0767619Z FINISHED PRINTING LOG FILE of distributed/test_c10d_gloo (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_c10d_gloo_rs8ndduz) 2022-11-23T03:14:27.0767631Z 2022-11-23T03:14:27.0768012Z Running distributed/test_c10d_common ... [2022-11-23 03:14:26.916638] 2022-11-23T03:14:27.0768515Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_common.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:14:26.917022] 2022-11-23T03:15:50.1150432Z 2022-11-23T03:15:50.1151371Z Expand the folded group to see the log file of distributed/test_c10d_common 2022-11-23T03:15:50.1153470Z ##[group]PRINTING LOG FILE of distributed/test_c10d_common (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_c10d_common_dhse4uky) 2022-11-23T03:15:50.1155179Z ]> 2022-11-23T03:15:50.1156495Z test_debug_level (__main__.CommTest) 2022-11-23T03:15:50.1159021Z , <__main__.ComputeBucketAssignmentTest testMethod=test_multi_limit_single_dtype>, <__main__.ComputeBucketAssignmentTest testMethod=test_single_limit_multi_dtype>, <__main__.ComputeBucketAssignmentTest testMethod=test_single_limit_single_dtype>]> 2022-11-23T03:15:50.1162158Z test_multi_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) 2022-11-23T03:15:50.1163479Z test_multi_limit_single_dtype (__main__.ComputeBucketAssignmentTest) 2022-11-23T03:15:50.1164884Z test_single_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) 2022-11-23T03:15:50.1166317Z test_single_limit_single_dtype (__main__.ComputeBucketAssignmentTest) 2022-11-23T03:15:50.1169296Z , <__main__.PythonProcessGroupExtensionTest testMethod=test_collectives>, <__main__.PythonProcessGroupExtensionTest testMethod=test_get_backend_name>, <__main__.PythonProcessGroupExtensionTest testMethod=test_send_recv>]> 2022-11-23T03:15:50.1171958Z test_backend_class_attr (__main__.PythonProcessGroupExtensionTest) 2022-11-23T03:15:50.1173856Z test_collectives (__main__.PythonProcessGroupExtensionTest) 2022-11-23T03:15:50.1175256Z test_get_backend_name (__main__.PythonProcessGroupExtensionTest) 2022-11-23T03:15:50.1176694Z test_send_recv (__main__.PythonProcessGroupExtensionTest) 2022-11-23T03:15:50.1179282Z , <__main__.ReduceOpTest testMethod=test_reduceop_copyable>, <__main__.ReduceOpTest testMethod=test_reduceop_pickle>]> 2022-11-23T03:15:50.1181220Z test_op_isinstance_of_reduceop (__main__.ReduceOpTest) 2022-11-23T03:15:50.1182347Z test_reduceop_copyable (__main__.ReduceOpTest) 2022-11-23T03:15:50.1183398Z test_reduceop_pickle (__main__.ReduceOpTest) 2022-11-23T03:15:50.1185207Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-11-23T03:15:50.1187062Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:15:50.1188358Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:15:50.1190050Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:15:50.1191364Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:15:50.1191988Z 2022-11-23T03:15:50.1192280Z Running tests... 2022-11-23T03:15:50.1193488Z ---------------------------------------------------------------------- 2022-11-23T03:15:50.1194840Z test_debug_level (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2405 2022-11-23T03:15:50.1196218Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2406 2022-11-23T03:15:50.1197983Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:15:50.1199244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:15:50.1200943Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:15:50.1202264Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:15:50.1203482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:15:50.1205266Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:15:50.1206498Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:15:50.1208338Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:15:50.1209656Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:15:50.1210873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:15:50.1211846Z ok (5.064s) 2022-11-23T03:15:50.1212482Z 2022-11-23T03:15:50.1213318Z ---------------------------------------------------------------------- 2022-11-23T03:15:50.1214242Z Ran 1 test in 5.065s 2022-11-23T03:15:50.1214651Z 2022-11-23T03:15:50.1214897Z OK 2022-11-23T03:15:50.1215256Z 2022-11-23T03:15:50.1215583Z Generating XML reports... 2022-11-23T03:15:50.1217183Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-CommTest-20221123031430.xml 2022-11-23T03:15:50.1218921Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-11-23T03:15:50.1220713Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:15:50.1221954Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:15:50.1223614Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:15:50.1225146Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:15:50.1225797Z 2022-11-23T03:15:50.1226093Z Running tests... 2022-11-23T03:15:50.1227297Z ---------------------------------------------------------------------- 2022-11-23T03:15:50.1228505Z test_multi_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) ... ok (0.596s) 2022-11-23T03:15:50.1229201Z 2022-11-23T03:15:50.1229978Z ---------------------------------------------------------------------- 2022-11-23T03:15:50.1230884Z Ran 1 test in 0.596s 2022-11-23T03:15:50.1231310Z 2022-11-23T03:15:50.1231552Z OK 2022-11-23T03:15:50.1231884Z 2022-11-23T03:15:50.1232211Z Generating XML reports... 2022-11-23T03:15:50.1233988Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221123031439.xml 2022-11-23T03:15:50.1235876Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-11-23T03:15:50.1237672Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:15:50.1238936Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:15:50.1240612Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:15:50.1241914Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:15:50.1242530Z 2022-11-23T03:15:50.1242795Z Running tests... 2022-11-23T03:15:50.1243976Z ---------------------------------------------------------------------- 2022-11-23T03:15:50.1245189Z test_multi_limit_single_dtype (__main__.ComputeBucketAssignmentTest) ... ok (0.592s) 2022-11-23T03:15:50.1245881Z 2022-11-23T03:15:50.1246658Z ---------------------------------------------------------------------- 2022-11-23T03:15:50.1247568Z Ran 1 test in 0.593s 2022-11-23T03:15:50.1248540Z 2022-11-23T03:15:50.1248753Z OK 2022-11-23T03:15:50.1249072Z 2022-11-23T03:15:50.1249366Z Generating XML reports... 2022-11-23T03:15:50.1251007Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221123031444.xml 2022-11-23T03:15:50.1252747Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-11-23T03:15:50.1254386Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:15:50.1255517Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:15:50.1257054Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:15:50.1258240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:15:50.1258796Z 2022-11-23T03:15:50.1259026Z Running tests... 2022-11-23T03:15:50.1259524Z ---------------------------------------------------------------------- 2022-11-23T03:15:50.1259987Z test_single_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) ... ok (0.587s) 2022-11-23T03:15:50.1260312Z 2022-11-23T03:15:50.1260582Z ---------------------------------------------------------------------- 2022-11-23T03:15:50.1260896Z Ran 1 test in 0.587s 2022-11-23T03:15:50.1261044Z 2022-11-23T03:15:50.1261126Z OK 2022-11-23T03:15:50.1261247Z 2022-11-23T03:15:50.1261359Z Generating XML reports... 2022-11-23T03:15:50.1261972Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221123031449.xml 2022-11-23T03:15:50.1262610Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-11-23T03:15:50.1263229Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:15:50.1263660Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:15:50.1264232Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:15:50.1264747Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:15:50.1264965Z 2022-11-23T03:15:50.1265063Z Running tests... 2022-11-23T03:15:50.1265476Z ---------------------------------------------------------------------- 2022-11-23T03:15:50.1265884Z test_single_limit_single_dtype (__main__.ComputeBucketAssignmentTest) ... ok (0.588s) 2022-11-23T03:15:50.1266127Z 2022-11-23T03:15:50.1266391Z ---------------------------------------------------------------------- 2022-11-23T03:15:50.1266702Z Ran 1 test in 0.589s 2022-11-23T03:15:50.1266849Z 2022-11-23T03:15:50.1266931Z OK 2022-11-23T03:15:50.1267052Z 2022-11-23T03:15:50.1267166Z Generating XML reports... 2022-11-23T03:15:50.1267773Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221123031453.xml 2022-11-23T03:15:50.1268424Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-11-23T03:15:50.1269036Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:15:50.1269470Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:15:50.1270042Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:15:50.1270493Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:15:50.1270706Z 2022-11-23T03:15:50.1270806Z Running tests... 2022-11-23T03:15:50.1271210Z ---------------------------------------------------------------------- 2022-11-23T03:15:50.1271744Z test_backend_class_attr (__main__.PythonProcessGroupExtensionTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2870 2022-11-23T03:15:50.1272284Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2871 2022-11-23T03:15:50.1272708Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2872 2022-11-23T03:15:50.1273145Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2873 2022-11-23T03:15:50.1273749Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:15:50.1274180Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:15:50.1274749Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:15:50.1275201Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:15:50.1275621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:15:50.1276221Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:15:50.1276657Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:15:50.1277229Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:15:50.1277748Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:15:50.1278165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:15:50.1278781Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:15:50.1279211Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:15:50.1279772Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:15:50.1280223Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:15:50.1280643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:15:50.1281303Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:15:50.1281748Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:15:50.1282322Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:15:50.1282775Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:15:50.1283182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:15:50.1283519Z ok (5.441s) 2022-11-23T03:15:50.1283657Z 2022-11-23T03:15:50.1283923Z ---------------------------------------------------------------------- 2022-11-23T03:15:50.1284237Z Ran 1 test in 5.441s 2022-11-23T03:15:50.1284385Z 2022-11-23T03:15:50.1284467Z OK 2022-11-23T03:15:50.1284588Z 2022-11-23T03:15:50.1284701Z Generating XML reports... 2022-11-23T03:15:50.1285326Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221123031458.xml 2022-11-23T03:15:50.1285981Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-11-23T03:15:50.1286594Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:15:50.1287026Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:15:50.1287597Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:15:50.1288159Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:15:50.1288372Z 2022-11-23T03:15:50.1288471Z Running tests... 2022-11-23T03:15:50.1288880Z ---------------------------------------------------------------------- 2022-11-23T03:15:50.1289473Z test_collectives (__main__.PythonProcessGroupExtensionTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3205 2022-11-23T03:15:50.1290117Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3206 2022-11-23T03:15:50.1290638Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3207 2022-11-23T03:15:50.1291157Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3208 2022-11-23T03:15:50.1291878Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:15:50.1292391Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:15:50.1293073Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:15:50.1293611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:15:50.1294106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:15:50.1294874Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address). 2022-11-23T03:15:50.1295749Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:15:50.1296262Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:15:50.1296943Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:15:50.1297479Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:15:50.1297979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:15:50.1298724Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address). 2022-11-23T03:15:50.1299455Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:15:50.1299885Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:15:50.1300514Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:15:50.1300968Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:15:50.1301390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:15:50.1302055Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address). 2022-11-23T03:15:50.1302826Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:15:50.1303333Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:15:50.1304014Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:15:50.1304553Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:15:50.1305063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:15:50.1305616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:15:50.1306179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T03:15:50.1306737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T03:15:50.1307289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:15:50.1308069Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T03:15:50.1308871Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T03:15:50.1309690Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T03:15:50.1310497Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T03:15:50.1310945Z ok (6.346s) 2022-11-23T03:15:50.1311113Z 2022-11-23T03:15:50.1311432Z ---------------------------------------------------------------------- 2022-11-23T03:15:50.1311815Z Ran 1 test in 6.347s 2022-11-23T03:15:50.1311982Z 2022-11-23T03:15:50.1312080Z OK 2022-11-23T03:15:50.1312224Z 2022-11-23T03:15:50.1312358Z Generating XML reports... 2022-11-23T03:15:50.1313110Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221123031507.xml 2022-11-23T03:15:50.1313906Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-11-23T03:15:50.1314649Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:15:50.1315249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:15:50.1315941Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:15:50.1316468Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:15:50.1316725Z 2022-11-23T03:15:50.1316840Z Running tests... 2022-11-23T03:15:50.1317321Z ---------------------------------------------------------------------- 2022-11-23T03:15:50.1317955Z test_get_backend_name (__main__.PythonProcessGroupExtensionTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3545 2022-11-23T03:15:50.1318606Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3546 2022-11-23T03:15:50.1319048Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3547 2022-11-23T03:15:50.1319479Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3548 2022-11-23T03:15:50.1320126Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:15:50.1320563Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:15:50.1321280Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:15:50.1321731Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:15:50.1322135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:15:50.1322744Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:15:50.1323171Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:15:50.1323741Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:15:50.1324192Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:15:50.1324617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:15:50.1325231Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:15:50.1325651Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:15:50.1326221Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:15:50.1326669Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:15:50.1327089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:15:50.1327775Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:15:50.1328220Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:15:50.1328803Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:15:50.1329308Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:15:50.1329799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:15:50.1330197Z ok (4.734s) 2022-11-23T03:15:50.1330363Z 2022-11-23T03:15:50.1330682Z ---------------------------------------------------------------------- 2022-11-23T03:15:50.1331061Z Ran 1 test in 4.734s 2022-11-23T03:15:50.1331237Z 2022-11-23T03:15:50.1331337Z OK 2022-11-23T03:15:50.1331481Z 2022-11-23T03:15:50.1331620Z Generating XML reports... 2022-11-23T03:15:50.1332357Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221123031518.xml 2022-11-23T03:15:50.1333159Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-11-23T03:15:50.1334004Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:15:50.1334523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:15:50.1335217Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:15:50.1335760Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:15:50.1336020Z 2022-11-23T03:15:50.1336139Z Running tests... 2022-11-23T03:15:50.1336622Z ---------------------------------------------------------------------- 2022-11-23T03:15:50.1337225Z test_send_recv (__main__.PythonProcessGroupExtensionTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3880 2022-11-23T03:15:50.1337855Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3881 2022-11-23T03:15:50.1338375Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3882 2022-11-23T03:15:50.1338970Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3883 2022-11-23T03:15:50.1339608Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:15:50.1340041Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:15:50.1340615Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:15:50.1341055Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:15:50.1341485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:15:50.1342094Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:15:50.1342525Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:15:50.1343100Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:15:50.1343558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:15:50.1343976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:15:50.1344431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T03:15:50.1345052Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:15:50.1345479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:15:50.1346054Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:15:50.1346504Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:15:50.1346929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:15:50.1347395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:15:50.1348021Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:15:50.1348440Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:15:50.1349012Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:15:50.1349462Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:15:50.1349885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:15:50.1350356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T03:15:50.1350827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:15:50.1351574Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T03:15:50.1352238Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T03:15:50.1352913Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T03:15:50.1353597Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T03:15:50.1353977Z ok (5.433s) 2022-11-23T03:15:50.1354114Z 2022-11-23T03:15:50.1354379Z ---------------------------------------------------------------------- 2022-11-23T03:15:50.1354690Z Ran 1 test in 5.433s 2022-11-23T03:15:50.1354840Z 2022-11-23T03:15:50.1354925Z OK 2022-11-23T03:15:50.1355049Z 2022-11-23T03:15:50.1355153Z Generating XML reports... 2022-11-23T03:15:50.1355829Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221123031526.xml 2022-11-23T03:15:50.1356504Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-11-23T03:15:50.1357118Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:15:50.1357552Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:15:50.1358125Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:15:50.1358577Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:15:50.1358793Z 2022-11-23T03:15:50.1358890Z Running tests... 2022-11-23T03:15:50.1359286Z ---------------------------------------------------------------------- 2022-11-23T03:15:50.1359676Z test_op_isinstance_of_reduceop (__main__.ReduceOpTest) ... ok (0.592s) 2022-11-23T03:15:50.1359896Z 2022-11-23T03:15:50.1360167Z ---------------------------------------------------------------------- 2022-11-23T03:15:50.1360479Z Ran 1 test in 0.592s 2022-11-23T03:15:50.1360628Z 2022-11-23T03:15:50.1360715Z OK 2022-11-23T03:15:50.1360836Z 2022-11-23T03:15:50.1360951Z Generating XML reports... 2022-11-23T03:15:50.1361490Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20221123031536.xml 2022-11-23T03:15:50.1362098Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-11-23T03:15:50.1362714Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:15:50.1363149Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:15:50.1363725Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:15:50.1364175Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:15:50.1364398Z 2022-11-23T03:15:50.1364496Z Running tests... 2022-11-23T03:15:50.1364898Z ---------------------------------------------------------------------- 2022-11-23T03:15:50.1365268Z test_reduceop_copyable (__main__.ReduceOpTest) ... ok (0.591s) 2022-11-23T03:15:50.1365474Z 2022-11-23T03:15:50.1365738Z ---------------------------------------------------------------------- 2022-11-23T03:15:50.1366050Z Ran 1 test in 0.591s 2022-11-23T03:15:50.1366200Z 2022-11-23T03:15:50.1366283Z OK 2022-11-23T03:15:50.1366405Z 2022-11-23T03:15:50.1366518Z Generating XML reports... 2022-11-23T03:15:50.1367068Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20221123031540.xml 2022-11-23T03:15:50.1367664Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-11-23T03:15:50.1368411Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:15:50.1368924Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:15:50.1369567Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:15:50.1370108Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:15:50.1370365Z 2022-11-23T03:15:50.1370489Z Running tests... 2022-11-23T03:15:50.1370969Z ---------------------------------------------------------------------- 2022-11-23T03:15:50.1371402Z test_reduceop_pickle (__main__.ReduceOpTest) ... ok (0.598s) 2022-11-23T03:15:50.1371643Z 2022-11-23T03:15:50.1371962Z ---------------------------------------------------------------------- 2022-11-23T03:15:50.1372337Z Ran 1 test in 0.599s 2022-11-23T03:15:50.1372517Z 2022-11-23T03:15:50.1372616Z OK 2022-11-23T03:15:50.1372760Z 2022-11-23T03:15:50.1372896Z Generating XML reports... 2022-11-23T03:15:50.1373629Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20221123031545.xml 2022-11-23T03:15:50.1373997Z 2022-11-23T03:15:50.1374394Z ##[endgroup] 2022-11-23T03:15:50.1375063Z FINISHED PRINTING LOG FILE of distributed/test_c10d_common (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_c10d_common_dhse4uky) 2022-11-23T03:15:50.1375434Z 2022-11-23T03:15:50.1375785Z Running distributed/pipeline/sync/test_transparency ... [2022-11-23 03:15:50.115604] 2022-11-23T03:15:50.1376545Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_transparency.py', '-v'] ... [2022-11-23 03:15:50.116125] 2022-11-23T03:15:54.8570381Z 2022-11-23T03:15:54.8571735Z Expand the folded group to see the log file of distributed/pipeline/sync/test_transparency 2022-11-23T03:15:54.8574426Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_transparency (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_transparency_rxt9ec2x) 2022-11-23T03:15:54.8575879Z ============================= test session starts ============================== 2022-11-23T03:15:54.8577631Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T03:15:54.8578546Z cachedir: .pytest_cache 2022-11-23T03:15:54.8580100Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T03:15:54.8581329Z torch: 1.14.0a0+git1cfd385 2022-11-23T03:15:54.8582172Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T03:15:54.8583683Z plugins: shard-0.1.2, hypothesis-5.35.1, xdist-3.0.2, flakefinder-1.1.0, xdoctest-1.0.2, rerunfailures-10.3 2022-11-23T03:15:54.8584666Z collecting ... collected 1 item 2022-11-23T03:15:54.8585711Z Running 1 items in this shard: test/distributed/pipeline/sync/test_transparency.py::test_simple_linears 2022-11-23T03:15:54.8586415Z 2022-11-23T03:15:54.8587623Z distributed/pipeline/sync/test_transparency.py::test_simple_linears libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:15:54.8588716Z PASSED [100%] 2022-11-23T03:15:54.8589092Z 2022-11-23T03:15:54.8589468Z ============================== 1 passed in 0.16s =============================== 2022-11-23T03:15:54.8589944Z 2022-11-23T03:15:54.8590671Z ##[endgroup] 2022-11-23T03:15:54.8592435Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_transparency (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_transparency_rxt9ec2x) 2022-11-23T03:15:54.8593448Z 2022-11-23T03:15:54.8594221Z Running distributed/pipeline/sync/test_pipeline ... [2022-11-23 03:15:54.857304] 2022-11-23T03:15:54.8595861Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_pipeline.py', '-v'] ... [2022-11-23 03:15:54.857923] 2022-11-23T03:15:58.9542002Z 2022-11-23T03:15:58.9543587Z Expand the folded group to see the log file of distributed/pipeline/sync/test_pipeline 2022-11-23T03:15:58.9546406Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_pipeline (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_pipeline_lprx6cfg) 2022-11-23T03:15:58.9548446Z ============================= test session starts ============================== 2022-11-23T03:15:58.9550313Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T03:15:58.9551426Z cachedir: .pytest_cache 2022-11-23T03:15:58.9553053Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T03:15:58.9554319Z torch: 1.14.0a0+git1cfd385 2022-11-23T03:15:58.9555326Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T03:15:58.9557115Z plugins: shard-0.1.2, hypothesis-5.35.1, xdist-3.0.2, flakefinder-1.1.0, xdoctest-1.0.2, rerunfailures-10.3 2022-11-23T03:15:58.9558106Z collecting ... collected 1 item 2022-11-23T03:15:58.9559116Z Running 1 items in this shard: test/distributed/pipeline/sync/test_pipeline.py::test_clock_cycles 2022-11-23T03:15:58.9559816Z 2022-11-23T03:15:58.9560633Z distributed/pipeline/sync/test_pipeline.py::test_clock_cycles PASSED [100%] 2022-11-23T03:15:58.9561281Z 2022-11-23T03:15:58.9561660Z ============================== 1 passed in 0.02s =============================== 2022-11-23T03:15:58.9562127Z 2022-11-23T03:15:58.9562860Z ##[endgroup] 2022-11-23T03:15:58.9564590Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_pipeline (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_pipeline_lprx6cfg) 2022-11-23T03:15:58.9565551Z 2022-11-23T03:15:58.9566300Z Running distributed/pipeline/sync/test_phony ... [2022-11-23 03:15:58.954581] 2022-11-23T03:15:58.9568087Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_phony.py', '-v'] ... [2022-11-23 03:15:58.955191] 2022-11-23T03:16:03.0951837Z 2022-11-23T03:16:03.0952937Z Expand the folded group to see the log file of distributed/pipeline/sync/test_phony 2022-11-23T03:16:03.0955615Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_phony (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_phony_p9xxghql) 2022-11-23T03:16:03.0957032Z ============================= test session starts ============================== 2022-11-23T03:16:03.0958533Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T03:16:03.0959408Z cachedir: .pytest_cache 2022-11-23T03:16:03.0960909Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T03:16:03.0962018Z torch: 1.14.0a0+git1cfd385 2022-11-23T03:16:03.0962846Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T03:16:03.0964317Z plugins: shard-0.1.2, hypothesis-5.35.1, xdist-3.0.2, flakefinder-1.1.0, xdoctest-1.0.2, rerunfailures-10.3 2022-11-23T03:16:03.0965312Z collecting ... collected 4 items 2022-11-23T03:16:03.0967015Z Running 4 items in this shard: test/distributed/pipeline/sync/test_phony.py::test_phony_size, test/distributed/pipeline/sync/test_phony.py::test_phony_requires_grad, test/distributed/pipeline/sync/test_phony.py::test_cached_phony, test/distributed/pipeline/sync/test_phony.py::test_phony_in_autograd_function 2022-11-23T03:16:03.0968583Z 2022-11-23T03:16:03.0969102Z distributed/pipeline/sync/test_phony.py::test_phony_size PASSED [ 25%] 2022-11-23T03:16:03.0970201Z distributed/pipeline/sync/test_phony.py::test_phony_requires_grad PASSED [ 50%] 2022-11-23T03:16:03.0971318Z distributed/pipeline/sync/test_phony.py::test_cached_phony PASSED [ 75%] 2022-11-23T03:16:03.0972453Z distributed/pipeline/sync/test_phony.py::test_phony_in_autograd_function PASSED [100%] 2022-11-23T03:16:03.0973098Z 2022-11-23T03:16:03.0973471Z ============================== 4 passed in 0.05s =============================== 2022-11-23T03:16:03.0973939Z 2022-11-23T03:16:03.0974666Z ##[endgroup] 2022-11-23T03:16:03.0976341Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_phony (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_phony_p9xxghql) 2022-11-23T03:16:03.0977754Z 2022-11-23T03:16:03.0978527Z Running distributed/pipeline/sync/test_inplace ... [2022-11-23 03:16:03.095482] 2022-11-23T03:16:03.0980143Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_inplace.py', '-v'] ... [2022-11-23 03:16:03.096100] 2022-11-23T03:16:07.3331732Z 2022-11-23T03:16:07.3332617Z Expand the folded group to see the log file of distributed/pipeline/sync/test_inplace 2022-11-23T03:16:07.3334772Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_inplace (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_inplace_5am9koh7) 2022-11-23T03:16:07.3336160Z ============================= test session starts ============================== 2022-11-23T03:16:07.3337715Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T03:16:07.3338595Z cachedir: .pytest_cache 2022-11-23T03:16:07.3340603Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T03:16:07.3341760Z torch: 1.14.0a0+git1cfd385 2022-11-23T03:16:07.3342585Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T03:16:07.3344076Z plugins: shard-0.1.2, hypothesis-5.35.1, xdist-3.0.2, flakefinder-1.1.0, xdoctest-1.0.2, rerunfailures-10.3 2022-11-23T03:16:07.3345078Z collecting ... collected 3 items 2022-11-23T03:16:07.3346693Z Running 3 items in this shard: test/distributed/pipeline/sync/test_inplace.py::test_inplace_on_requires_grad, test/distributed/pipeline/sync/test_inplace.py::test_inplace_on_not_requires_grad, test/distributed/pipeline/sync/test_inplace.py::test_inplace_incorrect_grad 2022-11-23T03:16:07.3347939Z 2022-11-23T03:16:07.3349147Z distributed/pipeline/sync/test_inplace.py::test_inplace_on_requires_grad libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:16:07.3350229Z PASSED [ 33%] 2022-11-23T03:16:07.3351829Z distributed/pipeline/sync/test_inplace.py::test_inplace_on_not_requires_grad libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:16:07.3352926Z XFAIL [ 66%] 2022-11-23T03:16:07.3354481Z distributed/pipeline/sync/test_inplace.py::test_inplace_incorrect_grad libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:16:07.3355552Z XFAIL [100%] 2022-11-23T03:16:07.3355919Z 2022-11-23T03:16:07.3356317Z ========================= 1 passed, 2 xfailed in 0.19s ========================= 2022-11-23T03:16:07.3356805Z 2022-11-23T03:16:07.3357515Z ##[endgroup] 2022-11-23T03:16:07.3359215Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_inplace (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_inplace_5am9koh7) 2022-11-23T03:16:07.3360166Z 2022-11-23T03:16:07.3361019Z Running distributed/pipeline/sync/test_deferred_batch_norm ... [2022-11-23 03:16:07.333611] 2022-11-23T03:16:07.3362776Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_deferred_batch_norm.py', '-v'] ... [2022-11-23 03:16:07.334257] 2022-11-23T03:16:12.2613222Z 2022-11-23T03:16:12.2614761Z Expand the folded group to see the log file of distributed/pipeline/sync/test_deferred_batch_norm 2022-11-23T03:16:12.2617687Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_deferred_batch_norm (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_deferred_batch_norm_w3c12uga) 2022-11-23T03:16:12.2619126Z ============================= test session starts ============================== 2022-11-23T03:16:12.2621577Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T03:16:12.2622655Z cachedir: .pytest_cache 2022-11-23T03:16:12.2624266Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T03:16:12.2625382Z torch: 1.14.0a0+git1cfd385 2022-11-23T03:16:12.2626848Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T03:16:12.2628380Z plugins: shard-0.1.2, hypothesis-5.35.1, xdist-3.0.2, flakefinder-1.1.0, xdoctest-1.0.2, rerunfailures-10.3 2022-11-23T03:16:12.2629398Z collecting ... collected 11 items 2022-11-23T03:16:12.2635862Z Running 11 items in this shard: test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-1], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-4], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-1], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-4], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[0.1], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[None], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_convert_deferred_batch_norm, test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_eval, test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_optimize, test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_conv_bn, test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_input_requiring_grad 2022-11-23T03:16:12.2639357Z 2022-11-23T03:16:12.2640289Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-1] PASSED [ 9%] 2022-11-23T03:16:12.2641862Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-4] PASSED [ 18%] 2022-11-23T03:16:12.2643427Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-1] PASSED [ 27%] 2022-11-23T03:16:12.2644999Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-4] PASSED [ 36%] 2022-11-23T03:16:12.2646227Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[0.1] PASSED [ 45%] 2022-11-23T03:16:12.2647446Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[None] PASSED [ 54%] 2022-11-23T03:16:12.2648984Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_convert_deferred_batch_norm PASSED [ 63%] 2022-11-23T03:16:12.2650178Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_eval PASSED [ 72%] 2022-11-23T03:16:12.2651354Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_optimize PASSED [ 81%] 2022-11-23T03:16:12.2652515Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_conv_bn PASSED [ 90%] 2022-11-23T03:16:12.2653714Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_input_requiring_grad PASSED [100%] 2022-11-23T03:16:12.2654388Z 2022-11-23T03:16:12.2654762Z ============================== 11 passed in 0.95s ============================== 2022-11-23T03:16:12.2655226Z 2022-11-23T03:16:12.2655974Z ##[endgroup] 2022-11-23T03:16:12.2657841Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_deferred_batch_norm (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_deferred_batch_norm_w3c12uga) 2022-11-23T03:16:12.2658871Z 2022-11-23T03:16:12.2659652Z Running distributed/pipeline/sync/test_checkpoint ... [2022-11-23 03:16:12.261764] 2022-11-23T03:16:12.2661331Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_checkpoint.py', '-v'] ... [2022-11-23 03:16:12.262399] 2022-11-23T03:16:16.9067294Z 2022-11-23T03:16:16.9068510Z Expand the folded group to see the log file of distributed/pipeline/sync/test_checkpoint 2022-11-23T03:16:16.9071484Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_checkpoint (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_checkpoint_jvg3z_df) 2022-11-23T03:16:16.9073584Z ============================= test session starts ============================== 2022-11-23T03:16:16.9075345Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T03:16:16.9076400Z cachedir: .pytest_cache 2022-11-23T03:16:16.9078116Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T03:16:16.9079801Z torch: 1.14.0a0+git1cfd385 2022-11-23T03:16:16.9080710Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T03:16:16.9082378Z plugins: shard-0.1.2, hypothesis-5.35.1, xdist-3.0.2, flakefinder-1.1.0, xdoctest-1.0.2, rerunfailures-10.3 2022-11-23T03:16:16.9083495Z collecting ... collected 9 items 2022-11-23T03:16:16.9087673Z Running 9 items in this shard: test/distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cpu], test/distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cuda], test/distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad, test/distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad_with_parameter, test/distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cpu], test/distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cuda], test/distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing, test/distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing_without_checkpoint, test/distributed/pipeline/sync/test_checkpoint.py::test_non_grad_output 2022-11-23T03:16:16.9091799Z 2022-11-23T03:16:16.9092526Z distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cpu] PASSED [ 11%] 2022-11-23T03:16:16.9094033Z distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cuda] PASSED [ 22%] 2022-11-23T03:16:16.9095936Z distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad PASSED [ 33%] 2022-11-23T03:16:16.9097344Z distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad_with_parameter PASSED [ 44%] 2022-11-23T03:16:16.9098740Z distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cpu] PASSED [ 55%] 2022-11-23T03:16:16.9100119Z distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cuda] PASSED [ 66%] 2022-11-23T03:16:16.9101558Z distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing PASSED [ 77%] 2022-11-23T03:16:16.9103106Z distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing_without_checkpoint PASSED [ 88%] 2022-11-23T03:16:16.9104562Z distributed/pipeline/sync/test_checkpoint.py::test_non_grad_output PASSED [100%] 2022-11-23T03:16:16.9105268Z 2022-11-23T03:16:16.9105684Z ============================== 9 passed in 0.58s =============================== 2022-11-23T03:16:16.9106205Z 2022-11-23T03:16:16.9107030Z ##[endgroup] 2022-11-23T03:16:16.9109028Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_checkpoint (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_checkpoint_jvg3z_df) 2022-11-23T03:16:16.9110115Z 2022-11-23T03:16:16.9110932Z Running distributed/pipeline/sync/test_balance ... [2022-11-23 03:16:16.907176] 2022-11-23T03:16:16.9112770Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_balance.py', '-v'] ... [2022-11-23 03:16:16.907790] 2022-11-23T03:16:28.3783328Z 2022-11-23T03:16:28.3784561Z Expand the folded group to see the log file of distributed/pipeline/sync/test_balance 2022-11-23T03:16:28.3787554Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_balance (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_balance_90qw8h0z) 2022-11-23T03:16:28.3789286Z ============================= test session starts ============================== 2022-11-23T03:16:28.3790833Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T03:16:28.3791743Z cachedir: .pytest_cache 2022-11-23T03:16:28.3793279Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T03:16:28.3794414Z torch: 1.14.0a0+git1cfd385 2022-11-23T03:16:28.3795252Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T03:16:28.3796746Z plugins: shard-0.1.2, hypothesis-5.35.1, xdist-3.0.2, flakefinder-1.1.0, xdoctest-1.0.2, rerunfailures-10.3 2022-11-23T03:16:28.3798237Z collecting ... collected 18 items 2022-11-23T03:16:28.3804204Z Running 18 items in this shard: test/distributed/pipeline/sync/test_balance.py::test_blockpartition, test/distributed/pipeline/sync/test_balance.py::test_blockpartition_zeros, test/distributed/pipeline/sync/test_balance.py::test_blockpartition_non_positive_partitions, test/distributed/pipeline/sync/test_balance.py::test_blockpartition_short_sequence, test/distributed/pipeline/sync/test_balance.py::test_balance_by_time[cpu], test/distributed/pipeline/sync/test_balance.py::test_balance_by_time[cuda], test/distributed/pipeline/sync/test_balance.py::test_balance_by_time_loop_resets_input, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_latent, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_param, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_param_scale, test/distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cpu], test/distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cuda], test/distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cpu], test/distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cuda], test/distributed/pipeline/sync/test_balance.py::test_not_training, test/distributed/pipeline/sync/test_balance.py::test_balance_by_time_tuple, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_tuple, test/distributed/pipeline/sync/test_balance.py::test_already_has_grad 2022-11-23T03:16:28.3810390Z 2022-11-23T03:16:28.3811010Z distributed/pipeline/sync/test_balance.py::test_blockpartition PASSED [ 5%] 2022-11-23T03:16:28.3812325Z distributed/pipeline/sync/test_balance.py::test_blockpartition_zeros PASSED [ 11%] 2022-11-23T03:16:28.3813741Z distributed/pipeline/sync/test_balance.py::test_blockpartition_non_positive_partitions PASSED [ 16%] 2022-11-23T03:16:28.3815172Z distributed/pipeline/sync/test_balance.py::test_blockpartition_short_sequence PASSED [ 22%] 2022-11-23T03:16:28.3816550Z distributed/pipeline/sync/test_balance.py::test_balance_by_time[cpu] SKIPPED [ 27%] 2022-11-23T03:16:28.3817850Z distributed/pipeline/sync/test_balance.py::test_balance_by_time[cuda] SKIPPED [ 33%] 2022-11-23T03:16:28.3819185Z distributed/pipeline/sync/test_balance.py::test_balance_by_time_loop_resets_input PASSED [ 38%] 2022-11-23T03:16:28.3820828Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_latent PASSED [ 44%] 2022-11-23T03:16:28.3822270Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_param PASSED [ 50%] 2022-11-23T03:16:28.3823464Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_param_scale PASSED [ 55%] 2022-11-23T03:16:28.3824631Z distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cpu] PASSED [ 61%] 2022-11-23T03:16:28.3825815Z distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cuda] PASSED [ 66%] 2022-11-23T03:16:28.3827021Z distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cpu] PASSED [ 72%] 2022-11-23T03:16:28.3828273Z distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cuda] PASSED [ 77%] 2022-11-23T03:16:28.3829447Z distributed/pipeline/sync/test_balance.py::test_not_training PASSED [ 83%] 2022-11-23T03:16:28.3830597Z distributed/pipeline/sync/test_balance.py::test_balance_by_time_tuple PASSED [ 88%] 2022-11-23T03:16:28.3831757Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_tuple PASSED [ 94%] 2022-11-23T03:16:28.3832871Z distributed/pipeline/sync/test_balance.py::test_already_has_grad PASSED [100%] 2022-11-23T03:16:28.3833489Z 2022-11-23T03:16:28.3833891Z ======================== 16 passed, 2 skipped in 7.02s ========================= 2022-11-23T03:16:28.3834386Z 2022-11-23T03:16:28.3835151Z ##[endgroup] 2022-11-23T03:16:28.3836925Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_balance (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_balance_90qw8h0z) 2022-11-23T03:16:28.3838122Z 2022-11-23T03:16:28.3838958Z Running distributed/pipeline/sync/skip/test_tracker ... [2022-11-23 03:16:28.378802] 2022-11-23T03:16:28.3840647Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_tracker.py', '-v'] ... [2022-11-23 03:16:28.379389] 2022-11-23T03:16:33.3867340Z 2022-11-23T03:16:33.3868046Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_tracker 2022-11-23T03:16:33.3870688Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_tracker (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-skip-test_tracker__f9b3d3m) 2022-11-23T03:16:33.3872444Z ============================= test session starts ============================== 2022-11-23T03:16:33.3874446Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T03:16:33.3875584Z cachedir: .pytest_cache 2022-11-23T03:16:33.3877857Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T03:16:33.3879294Z torch: 1.14.0a0+git1cfd385 2022-11-23T03:16:33.3880342Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T03:16:33.3882147Z plugins: shard-0.1.2, hypothesis-5.35.1, xdist-3.0.2, flakefinder-1.1.0, xdoctest-1.0.2, rerunfailures-10.3 2022-11-23T03:16:33.3883164Z collecting ... collected 6 items 2022-11-23T03:16:33.3885685Z Running 6 items in this shard: test/distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker, test/distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker_by_data_parallel, test/distributed/pipeline/sync/skip/test_tracker.py::test_reuse_portal, test/distributed/pipeline/sync/skip/test_tracker.py::test_no_copy_no_portal, test/distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_without_checkpointing, test/distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_with_checkpointing 2022-11-23T03:16:33.3888062Z 2022-11-23T03:16:33.3888650Z distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker PASSED [ 16%] 2022-11-23T03:16:33.3889931Z distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker_by_data_parallel PASSED [ 33%] 2022-11-23T03:16:33.3891160Z distributed/pipeline/sync/skip/test_tracker.py::test_reuse_portal PASSED [ 50%] 2022-11-23T03:16:33.3892301Z distributed/pipeline/sync/skip/test_tracker.py::test_no_copy_no_portal PASSED [ 66%] 2022-11-23T03:16:33.3893551Z distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_without_checkpointing PASSED [ 83%] 2022-11-23T03:16:33.3894854Z distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_with_checkpointing PASSED [100%] 2022-11-23T03:16:33.3895547Z 2022-11-23T03:16:33.3895926Z ============================== 6 passed in 0.42s =============================== 2022-11-23T03:16:33.3896393Z 2022-11-23T03:16:33.3897136Z ##[endgroup] 2022-11-23T03:16:33.3898971Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_tracker (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-skip-test_tracker__f9b3d3m) 2022-11-23T03:16:33.3899992Z 2022-11-23T03:16:33.3900789Z Running distributed/pipeline/sync/skip/test_portal ... [2022-11-23 03:16:33.387084] 2022-11-23T03:16:33.3902479Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_portal.py', '-v'] ... [2022-11-23 03:16:33.387648] 2022-11-23T03:16:37.8108808Z 2022-11-23T03:16:37.8109872Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_portal 2022-11-23T03:16:37.8112834Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_portal (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-skip-test_portal_cids_20a) 2022-11-23T03:16:37.8114580Z ============================= test session starts ============================== 2022-11-23T03:16:37.8116250Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T03:16:37.8117362Z cachedir: .pytest_cache 2022-11-23T03:16:37.8119738Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T03:16:37.8120867Z torch: 1.14.0a0+git1cfd385 2022-11-23T03:16:37.8121892Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T03:16:37.8123749Z plugins: shard-0.1.2, hypothesis-5.35.1, xdist-3.0.2, flakefinder-1.1.0, xdoctest-1.0.2, rerunfailures-10.3 2022-11-23T03:16:37.8124947Z collecting ... collected 10 items 2022-11-23T03:16:37.8129221Z Running 10 items in this shard: test/distributed/pipeline/sync/skip/test_portal.py::test_copy_returns_on_next_device, test/distributed/pipeline/sync/skip/test_portal.py::test_blue_orange, test/distributed/pipeline/sync/skip/test_portal.py::test_blue_orange_not_requires_grad, test/distributed/pipeline/sync/skip/test_portal.py::test_use_grad, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_0, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_1, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_2, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_4, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3_plus_1 2022-11-23T03:16:37.8132362Z 2022-11-23T03:16:37.8132958Z distributed/pipeline/sync/skip/test_portal.py::test_copy_returns_on_next_device PASSED [ 10%] 2022-11-23T03:16:37.8134146Z distributed/pipeline/sync/skip/test_portal.py::test_blue_orange PASSED [ 20%] 2022-11-23T03:16:37.8135331Z distributed/pipeline/sync/skip/test_portal.py::test_blue_orange_not_requires_grad PASSED [ 30%] 2022-11-23T03:16:37.8136497Z distributed/pipeline/sync/skip/test_portal.py::test_use_grad PASSED [ 40%] 2022-11-23T03:16:37.8137657Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_0 PASSED [ 50%] 2022-11-23T03:16:37.8138908Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_1 PASSED [ 60%] 2022-11-23T03:16:37.8140149Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_2 PASSED [ 70%] 2022-11-23T03:16:37.8141377Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3 PASSED [ 80%] 2022-11-23T03:16:37.8142610Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_4 PASSED [ 90%] 2022-11-23T03:16:37.8143872Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3_plus_1 PASSED [100%] 2022-11-23T03:16:37.8144577Z 2022-11-23T03:16:37.8144955Z ============================== 10 passed in 0.39s ============================== 2022-11-23T03:16:37.8145425Z 2022-11-23T03:16:37.8146149Z ##[endgroup] 2022-11-23T03:16:37.8147970Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_portal (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-skip-test_portal_cids_20a) 2022-11-23T03:16:37.8148978Z 2022-11-23T03:16:37.8149872Z Running distributed/pipeline/sync/skip/test_inspect_skip_layout ... [2022-11-23 03:16:37.811305] 2022-11-23T03:16:37.8151677Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_inspect_skip_layout.py', '-v'] ... [2022-11-23 03:16:37.811917] 2022-11-23T03:16:41.8529071Z 2022-11-23T03:16:41.8530312Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_inspect_skip_layout 2022-11-23T03:16:41.8533108Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_inspect_skip_layout (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-skip-test_inspect_skip_layout_w11q386o) 2022-11-23T03:16:41.8534923Z ============================= test session starts ============================== 2022-11-23T03:16:41.8536329Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T03:16:41.8537367Z cachedir: .pytest_cache 2022-11-23T03:16:41.8539314Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T03:16:41.8540423Z torch: 1.14.0a0+git1cfd385 2022-11-23T03:16:41.8541248Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T03:16:41.8542827Z plugins: shard-0.1.2, hypothesis-5.35.1, xdist-3.0.2, flakefinder-1.1.0, xdoctest-1.0.2, rerunfailures-10.3 2022-11-23T03:16:41.8543868Z collecting ... collected 6 items 2022-11-23T03:16:41.8546705Z Running 6 items in this shard: test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_no_skippables, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_inner_partition, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_adjoining_partitions, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_far_partitions, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_pop_2_from_different_partitions, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_namespace 2022-11-23T03:16:41.8548938Z 2022-11-23T03:16:41.8549533Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_no_skippables PASSED [ 16%] 2022-11-23T03:16:41.8550791Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_inner_partition PASSED [ 33%] 2022-11-23T03:16:41.8552067Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_adjoining_partitions PASSED [ 50%] 2022-11-23T03:16:41.8553337Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_far_partitions PASSED [ 66%] 2022-11-23T03:16:41.8554648Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_pop_2_from_different_partitions PASSED [ 83%] 2022-11-23T03:16:41.8555939Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_namespace PASSED [100%] 2022-11-23T03:16:41.8556599Z 2022-11-23T03:16:41.8556967Z ============================== 6 passed in 0.03s =============================== 2022-11-23T03:16:41.8557430Z 2022-11-23T03:16:41.8558168Z ##[endgroup] 2022-11-23T03:16:41.8560120Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_inspect_skip_layout (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-skip-test_inspect_skip_layout_w11q386o) 2022-11-23T03:16:41.8561209Z 2022-11-23T03:16:41.8561962Z Running distributed/pipeline/sync/skip/test_api ... [2022-11-23 03:16:41.853363] 2022-11-23T03:16:41.8563615Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_api.py', '-v'] ... [2022-11-23 03:16:41.854016] 2022-11-23T03:16:46.0269110Z 2022-11-23T03:16:46.0269981Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_api 2022-11-23T03:16:46.0276572Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_api (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-skip-test_api_z_8zlhqc) 2022-11-23T03:16:46.0278202Z ============================= test session starts ============================== 2022-11-23T03:16:46.0280160Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T03:16:46.0281100Z cachedir: .pytest_cache 2022-11-23T03:16:46.0282960Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T03:16:46.0284319Z torch: 1.14.0a0+git1cfd385 2022-11-23T03:16:46.0285364Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T03:16:46.0286966Z plugins: shard-0.1.2, hypothesis-5.35.1, xdist-3.0.2, flakefinder-1.1.0, xdoctest-1.0.2, rerunfailures-10.3 2022-11-23T03:16:46.0288155Z collecting ... collected 3 items 2022-11-23T03:16:46.0289718Z Running 3 items in this shard: test/distributed/pipeline/sync/skip/test_api.py::test_namespace_difference, test/distributed/pipeline/sync/skip/test_api.py::test_namespace_copy, test/distributed/pipeline/sync/skip/test_api.py::test_skippable_repr 2022-11-23T03:16:46.0290905Z 2022-11-23T03:16:46.0291475Z distributed/pipeline/sync/skip/test_api.py::test_namespace_difference PASSED [ 33%] 2022-11-23T03:16:46.0293141Z distributed/pipeline/sync/skip/test_api.py::test_namespace_copy PASSED [ 66%] 2022-11-23T03:16:46.0294266Z distributed/pipeline/sync/skip/test_api.py::test_skippable_repr PASSED [100%] 2022-11-23T03:16:46.0294891Z 2022-11-23T03:16:46.0295273Z ============================== 3 passed in 0.03s =============================== 2022-11-23T03:16:46.0295746Z 2022-11-23T03:16:46.0296511Z ##[endgroup] 2022-11-23T03:16:46.0298295Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_api (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-skip-test_api_z_8zlhqc) 2022-11-23T03:16:46.0299270Z 2022-11-23T03:16:46.0300101Z Running distributed/optim/test_apply_optimizer_in_backward ... [2022-11-23 03:16:46.027390] 2022-11-23T03:16:46.0302129Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/optim/test_apply_optimizer_in_backward.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:16:46.027992] 2022-11-23T03:16:49.8426244Z 2022-11-23T03:16:49.8427362Z Expand the folded group to see the log file of distributed/optim/test_apply_optimizer_in_backward 2022-11-23T03:16:49.8430363Z ##[group]PRINTING LOG FILE of distributed/optim/test_apply_optimizer_in_backward (/var/lib/jenkins/pytorch/test/test-reports/distributed-optim-test_apply_optimizer_in_backward_6vs7mjf4) 2022-11-23T03:16:49.8431962Z 2022-11-23T03:16:49.8433023Z ##[endgroup] 2022-11-23T03:16:49.8436049Z FINISHED PRINTING LOG FILE of distributed/optim/test_apply_optimizer_in_backward (/var/lib/jenkins/pytorch/test/test-reports/distributed-optim-test_apply_optimizer_in_backward_6vs7mjf4) 2022-11-23T03:16:49.8437877Z 2022-11-23T03:16:49.8440052Z Running distributed/fsdp/test_wrap ... [2022-11-23 03:16:49.842924] 2022-11-23T03:16:49.8442764Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_wrap.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:16:49.843708] 2022-11-23T03:19:10.2192478Z 2022-11-23T03:19:10.2193514Z Expand the folded group to see the log file of distributed/fsdp/test_wrap 2022-11-23T03:19:10.2195560Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_wrap (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_wrap_d_hzssph) 2022-11-23T03:19:10.2197559Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_wrap 2022-11-23T03:19:10.2198272Z 2022-11-23T03:19:10.2198537Z Running tests... 2022-11-23T03:19:10.2199635Z ---------------------------------------------------------------------- 2022-11-23T03:19:10.2200558Z test_always_wrap (__main__.TestAutoWrap) 2022-11-23T03:19:10.2202593Z Test to ensure that if `always_wrap_policy` is ... ok (0.601s) 2022-11-23T03:19:10.2208547Z test_always_wrap_with_ignored_modules_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:10.2211915Z warnings.warn( 2022-11-23T03:19:10.2212608Z ok (0.005s) 2022-11-23T03:19:10.2213721Z test_always_wrap_with_ignored_modules_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... ok (0.004s) 2022-11-23T03:19:10.2214909Z test_auto_wrap_api (__main__.TestAutoWrap) 2022-11-23T03:19:10.2216297Z Test to ensure with auto wrap, we wrap child modules correctly based on the min_num_params. ... ok (0.003s) 2022-11-23T03:19:10.2218154Z test_auto_wrap_preset_exclude_wrap (__main__.TestAutoWrap) 2022-11-23T03:19:10.2219829Z Test to ensure excluded modules are not wrapped, regardless if the total param size is greater than the ... ok (0.002s) 2022-11-23T03:19:10.2221770Z test_auto_wrap_preset_exclude_wrap_include_children (__main__.TestAutoWrap) 2022-11-23T03:19:10.2224640Z Test to ensure excluded modules are not wrapped, but children are if param size is greater than ... ok (0.002s) 2022-11-23T03:19:10.2226348Z test_auto_wrap_preset_force_leaf (__main__.TestAutoWrap) 2022-11-23T03:19:10.2228654Z Test to ensure force-leaf modules are not wrapped, and children are not wrapped. The ... ok (0.003s) 2022-11-23T03:19:10.2230342Z test_auto_wrap_preset_force_leaf_custom (__main__.TestAutoWrap) 2022-11-23T03:19:10.2232148Z Test to ensure force-leaf modules are not wrapped. ... ok (0.002s) 2022-11-23T03:19:10.2234662Z test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=False)_use_device_id_False (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:10.2238040Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:19:10.2240626Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2243188Z ok (1.747s) 2022-11-23T03:19:10.2245327Z test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=False)_use_device_id_True (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:10.2248827Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:19:10.2252339Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:19:10.2256476Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:19:10.2258405Z ok (0.030s) 2022-11-23T03:19:10.2259813Z test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=True)_use_device_id_False (__main__.TestAutoWrap) ... ok (0.004s) 2022-11-23T03:19:10.2261831Z test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=True)_use_device_id_True (__main__.TestAutoWrap) ... ok (0.005s) 2022-11-23T03:19:10.2264225Z test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=False)_use_device_id_False (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:10.2266941Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:19:10.2268147Z ok (0.025s) 2022-11-23T03:19:10.2269946Z test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=False)_use_device_id_True (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:10.2272646Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:19:10.2273856Z ok (0.024s) 2022-11-23T03:19:10.2275636Z test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=True)_use_device_id_False (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:10.2278325Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:19:10.2282084Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:19:10.2286175Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:19:10.2290662Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:19:10.2294471Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:19:10.2296217Z ok (0.031s) 2022-11-23T03:19:10.2297850Z test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=True)_use_device_id_True (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:10.2299725Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:19:10.2300435Z ok (0.025s) 2022-11-23T03:19:10.2301116Z test_auto_wrap_with_ignored_modules_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... ok (0.004s) 2022-11-23T03:19:10.2302063Z test_auto_wrap_with_ignored_modules_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... ok (0.003s) 2022-11-23T03:19:10.2302830Z test_module_wrap_policy (__main__.TestAutoWrap) 2022-11-23T03:19:10.2304853Z Tests the ``ModuleWrapPolicy``. ... [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:19:10.2307393Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:19:10.2308506Z ok (0.019s) 2022-11-23T03:19:10.2309079Z test_transformer_auto_wrap_policy (__main__.TestAutoWrap) 2022-11-23T03:19:10.2309753Z Tests the ``transformer_auto_wrap_policy``. ... ok (0.018s) 2022-11-23T03:19:10.2310469Z test_wrap_disabled_outside_context (__main__.TestAutoWrap) ... ok (0.002s) 2022-11-23T03:19:10.2311197Z test_wrap_override_defaults (__main__.TestAutoWrap) ... ok (0.002s) 2022-11-23T03:19:10.2311963Z test_wrap_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... ok (0.002s) 2022-11-23T03:19:10.2312779Z test_wrap_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... ok (0.002s) 2022-11-23T03:19:10.2313531Z test_bn_always_wrapped_individually (__main__.TestFSDPWrap) 2022-11-23T03:19:10.2314487Z Ensures that by using _or_policy with _wrap_batchnorm_individually, even ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5499 2022-11-23T03:19:10.2315481Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5500 2022-11-23T03:19:10.2316658Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2317599Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2318718Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2319590Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2320410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:10.2321635Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2322472Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2323558Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2324523Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2325354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:10.2326604Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2328052Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2329144Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2330171Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2331031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:10.2331897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:10.2332562Z dist init r=1, world=2 2022-11-23T03:19:10.2334986Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:10.2336466Z warnings.warn( 2022-11-23T03:19:10.2336923Z dist init r=0, world=2 2022-11-23T03:19:10.2339320Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:10.2340773Z warnings.warn( 2022-11-23T03:19:10.2341208Z ok (4.462s) 2022-11-23T03:19:10.2341897Z test_error_already_wrapped_nested_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) 2022-11-23T03:19:10.2342940Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5642 2022-11-23T03:19:10.2343941Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5643 2022-11-23T03:19:10.2345120Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2345963Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2347072Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2347944Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2348919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:10.2350126Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2350936Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2352039Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2352906Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2353728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:10.2354978Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2356293Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2357483Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2358438Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2359282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:10.2360156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:10.2360821Z dist init r=0, world=2 2022-11-23T03:19:10.2363205Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:10.2364674Z warnings.warn( 2022-11-23T03:19:10.2365148Z dist init r=1, world=2 2022-11-23T03:19:10.2367516Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:10.2369220Z warnings.warn( 2022-11-23T03:19:10.2369658Z ok (4.345s) 2022-11-23T03:19:10.2370348Z test_error_already_wrapped_nested_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) 2022-11-23T03:19:10.2371426Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5785 2022-11-23T03:19:10.2372430Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5786 2022-11-23T03:19:10.2373639Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2374477Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2375595Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2376468Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2377270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:10.2378463Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2379130Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2379768Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2380306Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2380738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:10.2381387Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2382073Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2382624Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2383118Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2383555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:10.2384012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:10.2384367Z dist init r=1, world=2 2022-11-23T03:19:10.2384669Z dist init r=0, world=2 2022-11-23T03:19:10.2384895Z ok (4.449s) 2022-11-23T03:19:10.2385258Z test_error_already_wrapped_nested_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) 2022-11-23T03:19:10.2385811Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5928 2022-11-23T03:19:10.2386328Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5929 2022-11-23T03:19:10.2386946Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2387386Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2387966Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2388424Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2388854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:10.2389475Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2389915Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2390489Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2390946Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2391376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:10.2392024Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2392697Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2393262Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2393759Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2394200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:10.2394664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:10.2395009Z dist init r=0, world=2 2022-11-23T03:19:10.2396205Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:10.2397020Z warnings.warn( 2022-11-23T03:19:10.2397252Z dist init r=1, world=2 2022-11-23T03:19:10.2398434Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:10.2399178Z warnings.warn( 2022-11-23T03:19:10.2399411Z ok (4.442s) 2022-11-23T03:19:10.2399776Z test_error_already_wrapped_nested_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) 2022-11-23T03:19:10.2400329Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6071 2022-11-23T03:19:10.2400900Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6072 2022-11-23T03:19:10.2401528Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2401968Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2402533Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2402989Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2403427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:10.2404055Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2404494Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2405073Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2405530Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2405953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:10.2406601Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2407281Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2407903Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2408400Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2408841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:10.2409298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:10.2409714Z dist init r=0, world=2 2022-11-23T03:19:10.2409994Z dist init r=1, world=2 2022-11-23T03:19:10.2410282Z ok (4.539s) 2022-11-23T03:19:10.2411034Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6214 2022-11-23T03:19:10.2411886Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6215 2022-11-23T03:19:10.2412634Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2413160Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2413865Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2414407Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2415012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:10.2415775Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2416301Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2417011Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2417559Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2418077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:10.2418850Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2419687Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2420257Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2420756Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2421192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:10.2421652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:10.2421999Z dist init r=0, world=2 2022-11-23T03:19:10.2423184Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:10.2423958Z warnings.warn( 2022-11-23T03:19:10.2424193Z dist init r=1, world=2 2022-11-23T03:19:10.2425380Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:10.2426141Z warnings.warn( 2022-11-23T03:19:10.2426377Z ok (6.549s) 2022-11-23T03:19:10.2426998Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6367 2022-11-23T03:19:10.2427711Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6368 2022-11-23T03:19:10.2428330Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2428768Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2429343Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2429787Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2430222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:10.2430843Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2431278Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2431865Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2432418Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2432851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:10.2433486Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2434168Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2434728Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2435223Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2435658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:10.2436115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:10.2436527Z dist init r=1, world=2 2022-11-23T03:19:10.2436765Z dist init r=0, world=2 2022-11-23T03:19:10.2437002Z ok (6.548s) 2022-11-23T03:19:10.2437620Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6520 2022-11-23T03:19:10.2438316Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6521 2022-11-23T03:19:10.2438933Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2439367Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2439951Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2440416Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2440842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:10.2441463Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2441901Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2442475Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2442931Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2443361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:10.2444008Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2444694Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2445246Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2445743Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2446182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:10.2446639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:10.2446989Z dist init r=1, world=2 2022-11-23T03:19:10.2448290Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:10.2449119Z warnings.warn( 2022-11-23T03:19:10.2449393Z dist init r=0, world=2 2022-11-23T03:19:10.2450811Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:10.2451726Z warnings.warn( 2022-11-23T03:19:10.2452004Z ok (6.649s) 2022-11-23T03:19:10.2452750Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6673 2022-11-23T03:19:10.2453658Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6674 2022-11-23T03:19:10.2454406Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2454928Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2455618Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2456165Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2456674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:10.2457423Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2457946Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2458638Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2459197Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2459644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:10.2460289Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2460968Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2461519Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2462011Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2462445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:10.2462907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:10.2463259Z dist init r=0, world=2 2022-11-23T03:19:10.2463507Z dist init r=1, world=2 2022-11-23T03:19:10.2463730Z ok (6.756s) 2022-11-23T03:19:10.2464347Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6826 2022-11-23T03:19:10.2465048Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6827 2022-11-23T03:19:10.2465657Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2466101Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2466684Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2467204Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2467636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:10.2468247Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2468688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2469262Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2469717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2470145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:10.2470788Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2471514Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2472088Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2472568Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2473005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:10.2473461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:10.2473805Z dist init r=1, world=2 2022-11-23T03:19:10.2475001Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:10.2475754Z warnings.warn( 2022-11-23T03:19:10.2476000Z dist init r=0, world=2 2022-11-23T03:19:10.2477179Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:10.2477937Z warnings.warn( 2022-11-23T03:19:10.2478163Z ok (6.655s) 2022-11-23T03:19:10.2478783Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6979 2022-11-23T03:19:10.2479491Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6980 2022-11-23T03:19:10.2480101Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2480539Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2481118Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2481574Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2482002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:10.2482611Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2483054Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2483699Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2484155Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2484587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:10.2485234Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2485915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2486476Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2486958Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2487392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:10.2488039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:10.2488403Z dist init r=0, world=2 2022-11-23T03:19:10.2488653Z dist init r=1, world=2 2022-11-23T03:19:10.2488893Z ok (6.859s) 2022-11-23T03:19:10.2489541Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7132 2022-11-23T03:19:10.2490386Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7133 2022-11-23T03:19:10.2491129Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2491654Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2492345Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2492903Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2493429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:10.2494173Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2494682Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2495371Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2495918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2496434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:10.2497210Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2498040Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2498721Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2499307Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2499743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:10.2500199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:10.2500545Z dist init r=0, world=2 2022-11-23T03:19:10.2500793Z dist init r=1, world=2 2022-11-23T03:19:10.2501989Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:10.2502799Z warnings.warn( 2022-11-23T03:19:10.2503982Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:10.2504724Z warnings.warn( 2022-11-23T03:19:10.2504958Z ok (7.553s) 2022-11-23T03:19:10.2505609Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7285 2022-11-23T03:19:10.2506320Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7286 2022-11-23T03:19:10.2506934Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2507370Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2507948Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2508406Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2508840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:10.2509462Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2509884Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2510466Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2510924Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2511351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:10.2511995Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2512673Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2513234Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2513735Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2514162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:10.2514626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:10.2514974Z dist init r=0, world=2 2022-11-23T03:19:10.2515229Z dist init r=1, world=2 2022-11-23T03:19:10.2515468Z ok (6.650s) 2022-11-23T03:19:10.2516083Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7438 2022-11-23T03:19:10.2516777Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7439 2022-11-23T03:19:10.2517374Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2517814Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2518391Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2518911Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2519347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:10.2519978Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2520420Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2520985Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2521444Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2521878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:10.2522518Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2523255Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2523822Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2524317Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2524744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:10.2525206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:10.2525549Z dist init r=1, world=2 2022-11-23T03:19:10.2525795Z dist init r=0, world=2 2022-11-23T03:19:10.2526032Z ok (5.144s) 2022-11-23T03:19:10.2526659Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7581 2022-11-23T03:19:10.2527364Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7582 2022-11-23T03:19:10.2528062Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2528489Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2529072Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2529553Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2530068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:10.2530813Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2531335Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2532033Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2532575Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2533100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:10.2533877Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2534694Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2535373Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2535969Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2536499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:10.2537038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:10.2537547Z dist init r=0, world=2 2022-11-23T03:19:10.2537843Z dist init r=1, world=2 2022-11-23T03:19:10.2538127Z ok (6.749s) 2022-11-23T03:19:10.2538865Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7734 2022-11-23T03:19:10.2539684Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7735 2022-11-23T03:19:10.2540298Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2540723Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2541299Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2541811Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2542254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:10.2542879Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2543315Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2543889Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2544345Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2544766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:10.2545417Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2546098Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2546659Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2547153Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2547591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:10.2548059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:10.2548394Z dist init r=1, world=2 2022-11-23T03:19:10.2548643Z dist init r=0, world=2 2022-11-23T03:19:10.2548879Z ok (4.746s) 2022-11-23T03:19:10.2549489Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7877 2022-11-23T03:19:10.2550192Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7878 2022-11-23T03:19:10.2550812Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2551254Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2551819Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2552276Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2552714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:10.2553342Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2553778Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2554359Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2554880Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2555319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:10.2555961Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2556633Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2557193Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2557690Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2558121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:10.2558648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:10.2559002Z dist init r=1, world=2 2022-11-23T03:19:10.2559239Z dist init r=0, world=2 2022-11-23T03:19:10.2559477Z ok (7.151s) 2022-11-23T03:19:10.2560095Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8030 2022-11-23T03:19:10.2560790Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8031 2022-11-23T03:19:10.2561406Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2561846Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2562427Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2562878Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2563318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:10.2563937Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2564375Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2564952Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2565411Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2565840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:10.2566493Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2567162Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2567800Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2568351Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2568796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:10.2569255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:10.2569600Z dist init r=1, world=2 2022-11-23T03:19:10.2569851Z dist init r=0, world=2 2022-11-23T03:19:10.2570075Z ok (5.143s) 2022-11-23T03:19:10.2570685Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8173 2022-11-23T03:19:10.2571464Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8174 2022-11-23T03:19:10.2572084Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2572522Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2573099Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2573560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2573980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:10.2574604Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2575040Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2575687Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2576150Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2576586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:10.2577234Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2577913Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2578462Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2578958Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2579395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:10.2579851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:10.2580204Z dist init r=1, world=2 2022-11-23T03:19:10.2580454Z dist init r=0, world=2 2022-11-23T03:19:10.2580680Z ok (6.544s) 2022-11-23T03:19:10.2581290Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8326 2022-11-23T03:19:10.2581983Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8327 2022-11-23T03:19:10.2582593Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2583028Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2583606Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2584069Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2584507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:10.2585116Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2585552Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2586128Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2586587Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2587018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:10.2587664Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2588347Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2588987Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2589470Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2589906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:10.2590369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:10.2590718Z dist init r=1, world=2 2022-11-23T03:19:10.2590966Z dist init r=0, world=2 2022-11-23T03:19:10.2591204Z ok (4.644s) 2022-11-23T03:19:10.2591798Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8469 2022-11-23T03:19:10.2592541Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8470 2022-11-23T03:19:10.2593168Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2593612Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2594196Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2594653Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2595088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:10.2595712Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2596137Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2596716Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2597179Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2597611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:10.2598260Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2598943Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2599510Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2600006Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2600431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:10.2601005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:10.2601352Z dist init r=0, world=2 2022-11-23T03:19:10.2601603Z dist init r=1, world=2 2022-11-23T03:19:10.2601842Z ok (6.749s) 2022-11-23T03:19:10.2602293Z test_wrap_batchnorm_individually_use_or_policy_False (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8622 2022-11-23T03:19:10.2602836Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8623 2022-11-23T03:19:10.2603454Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2603877Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2604458Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2604918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2605352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:10.2606050Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2606483Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2607063Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2607521Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2608054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:10.2608699Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2609391Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2609955Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2610527Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2610970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:10.2611426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:10.2611761Z dist init r=0, world=2 2022-11-23T03:19:10.2612976Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:10.2613732Z warnings.warn( 2022-11-23T03:19:10.2613978Z dist init r=1, world=2 2022-11-23T03:19:10.2615169Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:10.2615919Z warnings.warn( 2022-11-23T03:19:10.2616152Z ok (5.044s) 2022-11-23T03:19:10.2616595Z test_wrap_batchnorm_individually_use_or_policy_True (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8765 2022-11-23T03:19:10.2617132Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8766 2022-11-23T03:19:10.2617742Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2618173Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2618751Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2619207Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2619639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:10.2620264Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:10.2620701Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:10.2621278Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:10.2621725Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:10.2622165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:10.2622885Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2623559Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:10.2624120Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2624617Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:10.2625052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:10.2625503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:10.2625851Z dist init r=1, world=2 2022-11-23T03:19:10.2627089Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:10.2627850Z warnings.warn( 2022-11-23T03:19:10.2628095Z dist init r=0, world=2 2022-11-23T03:19:10.2629288Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:10.2630041Z warnings.warn( 2022-11-23T03:19:10.2630280Z ok (5.043s) 2022-11-23T03:19:10.2630420Z 2022-11-23T03:19:10.2630707Z ---------------------------------------------------------------------- 2022-11-23T03:19:10.2631034Z Ran 47 tests in 135.985s 2022-11-23T03:19:10.2631177Z 2022-11-23T03:19:10.2631264Z OK 2022-11-23T03:19:10.2631391Z 2022-11-23T03:19:10.2631506Z Generating XML reports... 2022-11-23T03:19:10.2632072Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestAutoWrap-20221123031651.xml 2022-11-23T03:19:10.2632787Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestFSDPWrap-20221123031651.xml 2022-11-23T03:19:10.2633099Z 2022-11-23T03:19:10.2633544Z ##[endgroup] 2022-11-23T03:19:10.2634120Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_wrap (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_wrap_d_hzssph) 2022-11-23T03:19:10.2634442Z 2022-11-23T03:19:10.2634709Z Running distributed/fsdp/test_shard_utils ... [2022-11-23 03:19:10.221315] 2022-11-23T03:19:10.2635391Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_shard_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:19:10.221876] 2022-11-23T03:19:14.1808161Z 2022-11-23T03:19:14.1809256Z Expand the folded group to see the log file of distributed/fsdp/test_shard_utils 2022-11-23T03:19:14.1812107Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_shard_utils (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_shard_utils_k36fo521) 2022-11-23T03:19:14.1813002Z 2022-11-23T03:19:14.1813731Z ##[endgroup] 2022-11-23T03:19:14.1815592Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_shard_utils (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_shard_utils_k36fo521) 2022-11-23T03:19:14.1816482Z 2022-11-23T03:19:14.1823507Z Running distributed/fsdp/test_fsdp_uneven ... [2022-11-23 03:19:14.181273] 2022-11-23T03:19:14.1828871Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_uneven.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:19:14.182125] 2022-11-23T03:19:25.3868954Z 2022-11-23T03:19:25.3869957Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_uneven 2022-11-23T03:19:25.3873098Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_uneven (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_uneven_gbamflua) 2022-11-23T03:19:25.3875706Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven 2022-11-23T03:19:25.3876415Z 2022-11-23T03:19:25.3876678Z Running tests... 2022-11-23T03:19:25.3877817Z ---------------------------------------------------------------------- 2022-11-23T03:19:25.3878878Z test_one_iteration (__main__.TestUnevenParamShard) 2022-11-23T03:19:25.3880434Z Test FSDP with uneven divide of parameter shards. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9041 2022-11-23T03:19:25.3882075Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9042 2022-11-23T03:19:25.3884814Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:25.3886145Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:25.3888301Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:25.3889530Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:25.3890653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:25.3892325Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:25.3893465Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:25.3894994Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:25.3896202Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:25.3897351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:25.3899083Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:25.3900901Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:25.3902374Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:25.3903693Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:25.3904826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:25.3906033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:25.3906933Z dist init r=1, world=2 2022-11-23T03:19:25.3907763Z dist init r=0, world=2 2022-11-23T03:19:25.3908451Z ok (7.225s) 2022-11-23T03:19:25.3908882Z 2022-11-23T03:19:25.3909653Z ---------------------------------------------------------------------- 2022-11-23T03:19:25.3910485Z Ran 1 test in 7.225s 2022-11-23T03:19:25.3910881Z 2022-11-23T03:19:25.3911094Z OK 2022-11-23T03:19:25.3911414Z 2022-11-23T03:19:25.3911709Z Generating XML reports... 2022-11-23T03:19:25.3913319Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven/TEST-TestUnevenParamShard-20221123031915.xml 2022-11-23T03:19:25.3914213Z 2022-11-23T03:19:25.3914958Z ##[endgroup] 2022-11-23T03:19:25.3916562Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_uneven (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_uneven_gbamflua) 2022-11-23T03:19:25.3917453Z 2022-11-23T03:19:25.3918233Z Running distributed/fsdp/test_fsdp_tp_integration ... [2022-11-23 03:19:25.387212] 2022-11-23T03:19:25.3920143Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_tp_integration.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:19:25.387890] 2022-11-23T03:19:52.7578873Z 2022-11-23T03:19:52.7585321Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_tp_integration 2022-11-23T03:19:52.7587687Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_tp_integration (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_tp_integration_iz1h82fh) 2022-11-23T03:19:52.7589992Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_tp_integration 2022-11-23T03:19:52.7590747Z 2022-11-23T03:19:52.7591020Z Running tests... 2022-11-23T03:19:52.7592145Z ---------------------------------------------------------------------- 2022-11-23T03:19:52.7593221Z test_fsdp_tp_checkpoint_integration (__main__.TestTPFSDPIntegration) 2022-11-23T03:19:52.7594571Z Tests checkpointing for TP + FSDP integration. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9261 2022-11-23T03:19:52.7596454Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9262 2022-11-23T03:19:52.7598153Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:52.7599367Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:52.7600951Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:52.7602185Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:52.7603405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:52.7605100Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:52.7606279Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:52.7608242Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:52.7609489Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:52.7610612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:52.7612362Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:52.7614184Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:52.7615688Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:52.7617018Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:52.7618168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:52.7619576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:52.7620502Z dist init r=0, world=2 2022-11-23T03:19:52.7621140Z dist init r=1, world=2 2022-11-23T03:19:52.7621856Z skip: Need at least 4 CUDA devices (5.037s) 2022-11-23T03:19:52.7623018Z test_fsdp_tp_integration_tensor_parallel_size_2_cpu_offload_CPUOffload(offload_params=False) (__main__.TestTPFSDPIntegration) 2022-11-23T03:19:52.7625018Z Tests training for TP + FSDP integration by comparing an FSDP-only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9404 2022-11-23T03:19:52.7626401Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9405 2022-11-23T03:19:52.7628005Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:52.7629143Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:52.7630674Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:52.7632160Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:52.7633292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:52.7634965Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:52.7636096Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:52.7637616Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:52.7638808Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:52.7639945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:52.7641649Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:52.7643640Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:52.7645162Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:52.7646468Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:52.7647608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:52.7648948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:52.7649854Z dist init r=1, world=2 2022-11-23T03:19:52.7650472Z dist init r=0, world=2 2022-11-23T03:19:52.7651212Z skip: Need at least 4 CUDA devices (5.036s) 2022-11-23T03:19:52.7652365Z test_fsdp_tp_integration_tensor_parallel_size_2_cpu_offload_CPUOffload(offload_params=True) (__main__.TestTPFSDPIntegration) 2022-11-23T03:19:52.7654383Z Tests training for TP + FSDP integration by comparing an FSDP-only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9547 2022-11-23T03:19:52.7655778Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9548 2022-11-23T03:19:52.7657384Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:52.7658518Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:52.7660012Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:52.7661191Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:52.7662331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:52.7663978Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:52.7665107Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:52.7666640Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:52.7667820Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:52.7668950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:52.7670659Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:52.7672462Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:52.7673949Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:52.7675233Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:52.7676196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:52.7677378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:52.7678150Z dist init r=0, world=2 2022-11-23T03:19:52.7678674Z dist init r=1, world=2 2022-11-23T03:19:52.7679298Z skip: Need at least 4 CUDA devices (4.332s) 2022-11-23T03:19:52.7680274Z test_fsdp_tp_integration_tensor_parallel_size_4_cpu_offload_CPUOffload(offload_params=False) (__main__.TestTPFSDPIntegration) 2022-11-23T03:19:52.7681965Z Tests training for TP + FSDP integration by comparing an FSDP-only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9690 2022-11-23T03:19:52.7683128Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9691 2022-11-23T03:19:52.7684486Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:52.7685452Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:52.7686841Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:52.7688105Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:52.7689167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:52.7690842Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:52.7691971Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:52.7693500Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:52.7694687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:52.7695365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:52.7696160Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:52.7696989Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:52.7697667Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:52.7698257Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:52.7698781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:52.7699335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:52.7699755Z dist init r=0, world=2 2022-11-23T03:19:52.7700038Z dist init r=1, world=2 2022-11-23T03:19:52.7700385Z skip: Need at least 4 CUDA devices (4.433s) 2022-11-23T03:19:52.7700915Z test_fsdp_tp_integration_tensor_parallel_size_4_cpu_offload_CPUOffload(offload_params=True) (__main__.TestTPFSDPIntegration) 2022-11-23T03:19:52.7701816Z Tests training for TP + FSDP integration by comparing an FSDP-only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9833 2022-11-23T03:19:52.7702445Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9834 2022-11-23T03:19:52.7703187Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:52.7703717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:52.7704406Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:52.7704951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:52.7705478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:52.7706111Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:52.7706632Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:52.7707215Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:52.7707670Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:52.7708092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:52.7708744Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:52.7709422Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:52.7709978Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:52.7710474Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:52.7710976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:52.7711446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:52.7711787Z dist init r=1, world=2 2022-11-23T03:19:52.7712035Z dist init r=0, world=2 2022-11-23T03:19:52.7712319Z skip: Need at least 4 CUDA devices (4.432s) 2022-11-23T03:19:52.7712498Z 2022-11-23T03:19:52.7712774Z ---------------------------------------------------------------------- 2022-11-23T03:19:52.7713115Z Ran 5 tests in 23.272s 2022-11-23T03:19:52.7713299Z 2022-11-23T03:19:52.7713417Z OK (skipped=5) 2022-11-23T03:19:52.7713591Z 2022-11-23T03:19:52.7713726Z Generating XML reports... 2022-11-23T03:19:52.7714477Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_tp_integration/TEST-TestTPFSDPIntegration-20221123031927.xml 2022-11-23T03:19:52.7714908Z 2022-11-23T03:19:52.7715402Z ##[endgroup] 2022-11-23T03:19:52.7716174Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_tp_integration (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_tp_integration_iz1h82fh) 2022-11-23T03:19:52.7716608Z 2022-11-23T03:19:52.7716934Z Running distributed/fsdp/test_fsdp_state_dict ... [2022-11-23 03:19:52.758227] 2022-11-23T03:19:52.7717699Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_state_dict.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:19:52.758912] 2022-11-23T03:30:38.9458194Z 2022-11-23T03:30:38.9458982Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_state_dict 2022-11-23T03:30:38.9461147Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_state_dict (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_state_dict_e84dnls9) 2022-11-23T03:30:38.9465717Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict 2022-11-23T03:30:38.9466506Z 2022-11-23T03:30:38.9466769Z Running tests... 2022-11-23T03:30:38.9471141Z ---------------------------------------------------------------------- 2022-11-23T03:30:38.9473623Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9476316Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10043 2022-11-23T03:30:38.9477871Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10044 2022-11-23T03:30:38.9479789Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9481546Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9483864Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9486094Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9487432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9489995Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9491457Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9493480Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9495217Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9496664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9499005Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9501839Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9503844Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9505678Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9507162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9508728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9509968Z dist init r=1, world=2 2022-11-23T03:30:38.9510852Z dist init r=0, world=2 2022-11-23T03:30:38.9512492Z ok (5.385s) 2022-11-23T03:30:38.9514158Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9516543Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10190 2022-11-23T03:30:38.9518362Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10191 2022-11-23T03:30:38.9520645Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9522269Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9524328Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9525906Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9527447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9529945Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9531430Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9533506Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9535019Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9536521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9538769Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9541069Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9542970Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9544743Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9546248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9548111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9549243Z dist init r=0, world=2 2022-11-23T03:30:38.9550147Z dist init r=1, world=2 2022-11-23T03:30:38.9550968Z ok (4.630s) 2022-11-23T03:30:38.9552693Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9554921Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10333 2022-11-23T03:30:38.9556718Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10334 2022-11-23T03:30:38.9558889Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9560700Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9562926Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9564470Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9565960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9568275Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9569808Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9571838Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9573442Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9574899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9577134Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9579509Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9581165Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9582038Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9582806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9583529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9584072Z dist init r=1, world=2 2022-11-23T03:30:38.9584445Z dist init r=0, world=2 2022-11-23T03:30:38.9584809Z ok (4.529s) 2022-11-23T03:30:38.9585560Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9586556Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10476 2022-11-23T03:30:38.9587260Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10477 2022-11-23T03:30:38.9588017Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9588549Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9589246Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9589786Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9590315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9591185Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9591710Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9592408Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9592954Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9593475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9594247Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9595079Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9595762Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9596444Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9596973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9597536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9597951Z dist init r=1, world=2 2022-11-23T03:30:38.9598233Z dist init r=0, world=2 2022-11-23T03:30:38.9598519Z ok (4.829s) 2022-11-23T03:30:38.9599113Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9599915Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10619 2022-11-23T03:30:38.9600530Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10620 2022-11-23T03:30:38.9601281Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9601806Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9602495Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9603035Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9603571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9604324Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9604851Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9605551Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9606098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9606624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9607392Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9608374Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9609054Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9609653Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9610173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9610725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9611149Z dist init r=1, world=2 2022-11-23T03:30:38.9611534Z dist init r=0, world=2 2022-11-23T03:30:38.9611826Z ok (4.932s) 2022-11-23T03:30:38.9612428Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9613236Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10766 2022-11-23T03:30:38.9613856Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10767 2022-11-23T03:30:38.9614606Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9615138Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9615821Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9616440Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9616977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9617742Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9618266Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9618958Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9619511Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9620035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9620803Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9621641Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9622331Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9622932Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9623457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9624016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9624428Z dist init r=0, world=2 2022-11-23T03:30:38.9624719Z dist init r=1, world=2 2022-11-23T03:30:38.9625002Z ok (4.432s) 2022-11-23T03:30:38.9625593Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9626407Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10909 2022-11-23T03:30:38.9627030Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10910 2022-11-23T03:30:38.9627775Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9628307Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9628987Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9629539Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9630063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9630817Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9631347Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9632123Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9632672Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9633194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9633964Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9634783Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9635459Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9636060Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9636581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9637196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9637620Z dist init r=1, world=2 2022-11-23T03:30:38.9637904Z dist init r=0, world=2 2022-11-23T03:30:38.9638201Z ok (4.529s) 2022-11-23T03:30:38.9638795Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9639587Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11052 2022-11-23T03:30:38.9640206Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11053 2022-11-23T03:30:38.9640947Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9641471Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9642165Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9642713Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9643249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9644009Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9644528Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9645222Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9645769Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9646277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9647062Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9647952Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9648641Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9649233Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9649764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9650317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9650734Z dist init r=1, world=2 2022-11-23T03:30:38.9651023Z dist init r=0, world=2 2022-11-23T03:30:38.9651308Z ok (4.629s) 2022-11-23T03:30:38.9651904Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9652905Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11195 2022-11-23T03:30:38.9653527Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11196 2022-11-23T03:30:38.9654282Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9654813Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9655502Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9656053Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9656580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9657401Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9657938Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9658640Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9659195Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9659702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9660496Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9661318Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9661930Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9662434Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9662877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9663338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9663690Z dist init r=0, world=2 2022-11-23T03:30:38.9663928Z dist init r=1, world=2 2022-11-23T03:30:38.9664169Z ok (4.831s) 2022-11-23T03:30:38.9664661Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9665320Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11342 2022-11-23T03:30:38.9665839Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11343 2022-11-23T03:30:38.9666462Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9666900Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9667467Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9667926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9668362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9669000Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9669436Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9670016Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9670477Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9670960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9671618Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9672296Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9672858Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9673358Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9673793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9674257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9674610Z dist init r=1, world=2 2022-11-23T03:30:38.9674849Z dist init r=0, world=2 2022-11-23T03:30:38.9675163Z ok (4.529s) 2022-11-23T03:30:38.9675656Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9676329Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11485 2022-11-23T03:30:38.9676853Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11486 2022-11-23T03:30:38.9677473Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9677916Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9678484Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9678945Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9679386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9680016Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9680457Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9681037Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9681492Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9681916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9682564Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9683254Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9683826Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9684324Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9684759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9685218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9685554Z dist init r=1, world=2 2022-11-23T03:30:38.9685808Z dist init r=0, world=2 2022-11-23T03:30:38.9686053Z ok (4.529s) 2022-11-23T03:30:38.9686542Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9687207Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11628 2022-11-23T03:30:38.9687826Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11629 2022-11-23T03:30:38.9688458Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9688896Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9689545Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9690094Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9690617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9691371Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9691897Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9692678Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9693232Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9693738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9694525Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9695346Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9696026Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9696621Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9697140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9697705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9698116Z dist init r=1, world=2 2022-11-23T03:30:38.9698415Z dist init r=0, world=2 2022-11-23T03:30:38.9698705Z ok (4.429s) 2022-11-23T03:30:38.9699302Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9700096Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11771 2022-11-23T03:30:38.9700719Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11772 2022-11-23T03:30:38.9701460Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9701949Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9702531Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9702993Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9703430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9704061Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9704494Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9705070Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9705523Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9705945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9706599Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9707347Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9707910Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9708406Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9708844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9709303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9709641Z dist init r=1, world=2 2022-11-23T03:30:38.9709894Z dist init r=0, world=2 2022-11-23T03:30:38.9710135Z ok (5.039s) 2022-11-23T03:30:38.9710673Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9711332Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11918 2022-11-23T03:30:38.9711849Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11919 2022-11-23T03:30:38.9712469Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9712895Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9713469Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9713930Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9714365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9715003Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9715442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9716017Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9716470Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9716895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9717546Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9718225Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9718783Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9719289Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9719735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9720191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9720526Z dist init r=0, world=2 2022-11-23T03:30:38.9720776Z dist init r=1, world=2 2022-11-23T03:30:38.9721017Z ok (5.229s) 2022-11-23T03:30:38.9721503Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9722163Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12061 2022-11-23T03:30:38.9722672Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12062 2022-11-23T03:30:38.9723288Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9723774Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9724357Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9724812Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9725246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9725869Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9726304Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9726877Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9727335Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9727933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9728602Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9729310Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9729982Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9730573Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9731092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9731643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9732047Z dist init r=0, world=2 2022-11-23T03:30:38.9732346Z dist init r=1, world=2 2022-11-23T03:30:38.9732633Z ok (5.031s) 2022-11-23T03:30:38.9733235Z test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9734033Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12204 2022-11-23T03:30:38.9734646Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12205 2022-11-23T03:30:38.9735400Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9735920Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9736609Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9737167Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9737699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9738455Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9738976Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9739670Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9740204Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9740728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9741513Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9742228Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9742858Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9743353Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9743793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9744251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9744589Z dist init r=0, world=2 2022-11-23T03:30:38.9744844Z dist init r=1, world=2 2022-11-23T03:30:38.9745081Z ok (5.232s) 2022-11-23T03:30:38.9745584Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9746251Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12347 2022-11-23T03:30:38.9746820Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12348 2022-11-23T03:30:38.9747440Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9747865Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9748445Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9748901Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9749339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9749963Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9750402Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9750986Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9751431Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9751866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9752513Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9753187Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9753750Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9754246Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9754684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9755143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9755485Z dist init r=0, world=2 2022-11-23T03:30:38.9755736Z dist init r=1, world=2 2022-11-23T03:30:38.9755975Z ok (5.132s) 2022-11-23T03:30:38.9756476Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9757133Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12494 2022-11-23T03:30:38.9757645Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12495 2022-11-23T03:30:38.9758259Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9758690Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9759270Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9759786Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9760219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9760849Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9761285Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9761861Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9762311Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9762748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9763438Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9764126Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9764693Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9765189Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9765625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9766081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9766417Z dist init r=1, world=2 2022-11-23T03:30:38.9766668Z dist init r=0, world=2 2022-11-23T03:30:38.9766910Z ok (5.432s) 2022-11-23T03:30:38.9767414Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9768143Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12641 2022-11-23T03:30:38.9768655Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12642 2022-11-23T03:30:38.9769272Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9769792Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9770495Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9771043Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9771571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9772334Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9772863Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9773556Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9774088Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9774615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9775394Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9776218Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9776889Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9777491Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9778099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9778632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9779057Z dist init r=0, world=2 2022-11-23T03:30:38.9779354Z dist init r=1, world=2 2022-11-23T03:30:38.9779638Z ok (5.126s) 2022-11-23T03:30:38.9780239Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9781042Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12784 2022-11-23T03:30:38.9781661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12785 2022-11-23T03:30:38.9782360Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9782794Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9783378Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9783837Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9784273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9784899Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9785339Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9785914Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9786354Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9786798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9787448Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9788129Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9788693Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9789190Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9789626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9790079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9790426Z dist init r=1, world=2 2022-11-23T03:30:38.9790677Z dist init r=0, world=2 2022-11-23T03:30:38.9790922Z ok (5.332s) 2022-11-23T03:30:38.9791434Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9792103Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12927 2022-11-23T03:30:38.9792616Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12928 2022-11-23T03:30:38.9793213Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9793655Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9794243Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9794699Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9795201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9795827Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9796259Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9796834Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9797277Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9797714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9798362Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9799038Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9799656Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9800156Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9800590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9801034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9801388Z dist init r=0, world=2 2022-11-23T03:30:38.9801639Z dist init r=1, world=2 2022-11-23T03:30:38.9801881Z ok (5.732s) 2022-11-23T03:30:38.9802377Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9803039Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13074 2022-11-23T03:30:38.9803561Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13075 2022-11-23T03:30:38.9804157Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9804602Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9805185Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9805640Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9806075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9806696Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9807131Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9807843Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9808299Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9808737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9809417Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9810237Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9810922Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9811518Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9812042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9812581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9813105Z dist init r=1, world=2 2022-11-23T03:30:38.9813409Z dist init r=0, world=2 2022-11-23T03:30:38.9813697Z ok (5.633s) 2022-11-23T03:30:38.9814294Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9815104Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13221 2022-11-23T03:30:38.9815723Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13222 2022-11-23T03:30:38.9816456Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9816984Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9817768Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9818332Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9818859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9819621Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9820141Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9820825Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9821382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9821914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9822566Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9823241Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9823810Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9824308Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9824745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9825194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9825547Z dist init r=1, world=2 2022-11-23T03:30:38.9825797Z dist init r=0, world=2 2022-11-23T03:30:38.9826038Z ok (5.232s) 2022-11-23T03:30:38.9826535Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9827209Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13364 2022-11-23T03:30:38.9827724Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13365 2022-11-23T03:30:38.9828323Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9828764Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9829342Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9829804Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9830241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9830866Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9831376Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9831941Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9832399Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9832839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9833486Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9834166Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9834729Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9835226Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9835720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9836168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9836523Z dist init r=0, world=2 2022-11-23T03:30:38.9836775Z dist init r=1, world=2 2022-11-23T03:30:38.9837014Z ok (5.032s) 2022-11-23T03:30:38.9837524Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9838193Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13507 2022-11-23T03:30:38.9838706Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13508 2022-11-23T03:30:38.9839313Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9839762Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9840341Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9840800Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9841237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9841863Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9842302Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9842867Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9843327Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9843770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9844425Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9845101Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9845666Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9846160Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9846597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9847049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9847398Z dist init r=0, world=2 2022-11-23T03:30:38.9847652Z dist init r=1, world=2 2022-11-23T03:30:38.9847940Z ok (5.233s) 2022-11-23T03:30:38.9848441Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9849189Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13654 2022-11-23T03:30:38.9849709Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13655 2022-11-23T03:30:38.9850320Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9850757Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9851338Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9851793Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9852282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9852919Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9853362Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9853932Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9854391Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9854825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9855472Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9856158Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9856727Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9857227Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9857664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9858108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9858460Z dist init r=0, world=2 2022-11-23T03:30:38.9858710Z dist init r=1, world=2 2022-11-23T03:30:38.9858953Z ok (5.333s) 2022-11-23T03:30:38.9859449Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9860117Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13801 2022-11-23T03:30:38.9860637Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13802 2022-11-23T03:30:38.9861246Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9861689Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9862273Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9862730Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9863170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9863798Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9864243Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9864810Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9865341Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9865779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9866431Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9867107Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9869475Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9869978Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9870417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9870860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9871274Z dist init r=0, world=2 2022-11-23T03:30:38.9871529Z dist init r=1, world=2 2022-11-23T03:30:38.9871773Z ok (5.239s) 2022-11-23T03:30:38.9872299Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9873099Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13944 2022-11-23T03:30:38.9873716Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13945 2022-11-23T03:30:38.9874447Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9874974Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9875679Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9876234Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9876761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9877513Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9878041Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9878724Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9879273Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9879798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9880575Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9881417Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9882094Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9882616Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9883055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9883505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9883854Z dist init r=0, world=2 2022-11-23T03:30:38.9884108Z dist init r=1, world=2 2022-11-23T03:30:38.9884350Z ok (5.036s) 2022-11-23T03:30:38.9884854Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9885608Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14087 2022-11-23T03:30:38.9886123Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14088 2022-11-23T03:30:38.9886728Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9887167Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9887787Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9888247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9888687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9889310Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9889814Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9890387Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9890846Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9891286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9891936Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9892611Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9893174Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9893674Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9894102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9894567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9894919Z dist init r=1, world=2 2022-11-23T03:30:38.9895174Z dist init r=0, world=2 2022-11-23T03:30:38.9895414Z ok (5.633s) 2022-11-23T03:30:38.9895909Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9896583Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14234 2022-11-23T03:30:38.9897098Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14235 2022-11-23T03:30:38.9897696Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9898144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9898730Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9899192Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9899626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9900248Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9900687Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9901251Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9901711Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9902149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9902873Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9903554Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9904117Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9904615Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9905035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9905498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9905853Z dist init r=0, world=2 2022-11-23T03:30:38.9906107Z dist init r=1, world=2 2022-11-23T03:30:38.9906347Z ok (5.131s) 2022-11-23T03:30:38.9906891Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9907563Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14381 2022-11-23T03:30:38.9908061Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14382 2022-11-23T03:30:38.9908678Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9909119Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9909698Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9910156Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9910601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9911228Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9911668Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9912230Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9912692Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9913135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9913781Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9914465Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9915033Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9915538Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9915961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9916425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9916782Z dist init r=0, world=2 2022-11-23T03:30:38.9917038Z dist init r=1, world=2 2022-11-23T03:30:38.9917278Z ok (4.931s) 2022-11-23T03:30:38.9917774Z test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9918444Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14524 2022-11-23T03:30:38.9918946Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14525 2022-11-23T03:30:38.9919635Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9920075Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9920661Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9921123Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9921562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9922188Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9922629Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9923196Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9923706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9924149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9924803Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9925480Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9926050Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9926548Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9926971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9927436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9927917Z dist init r=0, world=2 2022-11-23T03:30:38.9928172Z dist init r=1, world=2 2022-11-23T03:30:38.9928415Z ok (4.832s) 2022-11-23T03:30:38.9928916Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9929574Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14667 2022-11-23T03:30:38.9930076Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14668 2022-11-23T03:30:38.9930698Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9931139Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9931722Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9932194Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9932633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9933259Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9933680Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9934259Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9934717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9935152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9935802Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9936564Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9937129Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9937625Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9938051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9938513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9938860Z dist init r=1, world=2 2022-11-23T03:30:38.9939115Z dist init r=0, world=2 2022-11-23T03:30:38.9939359Z ok (5.432s) 2022-11-23T03:30:38.9939849Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9940582Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14814 2022-11-23T03:30:38.9941091Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14815 2022-11-23T03:30:38.9941718Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9942158Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9942736Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9943192Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9943633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9944257Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9944686Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9945268Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9945734Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9946173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9946824Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9947501Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9948066Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9948561Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9948987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9949458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9949806Z dist init r=0, world=2 2022-11-23T03:30:38.9950058Z dist init r=1, world=2 2022-11-23T03:30:38.9950298Z ok (5.230s) 2022-11-23T03:30:38.9950789Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9951449Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14961 2022-11-23T03:30:38.9951949Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14962 2022-11-23T03:30:38.9952565Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9953011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9953661Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9954117Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9954556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9955186Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9955611Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9956189Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9956652Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9957086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9957808Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9958491Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9959055Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9959555Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9959982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9960440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9960789Z dist init r=0, world=2 2022-11-23T03:30:38.9961041Z dist init r=1, world=2 2022-11-23T03:30:38.9961285Z ok (5.830s) 2022-11-23T03:30:38.9961776Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9962424Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15108 2022-11-23T03:30:38.9962942Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15109 2022-11-23T03:30:38.9963560Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9964000Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9964578Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9965031Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9965467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9966101Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9966525Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9967102Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9967558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9968052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9968703Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9969379Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9969942Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9970506Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9970944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9971407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9971757Z dist init r=1, world=2 2022-11-23T03:30:38.9972007Z dist init r=0, world=2 2022-11-23T03:30:38.9972247Z ok (5.434s) 2022-11-23T03:30:38.9972736Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9973377Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15255 2022-11-23T03:30:38.9973897Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15256 2022-11-23T03:30:38.9974577Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9975024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9975609Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9976067Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9976500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9977127Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9977552Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9978129Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9978602Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9979033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9979679Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9980365Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9980929Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9981415Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9981855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9982315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9982668Z dist init r=1, world=2 2022-11-23T03:30:38.9982923Z dist init r=0, world=2 2022-11-23T03:30:38.9983171Z ok (5.433s) 2022-11-23T03:30:38.9983660Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9984306Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15402 2022-11-23T03:30:38.9984821Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15403 2022-11-23T03:30:38.9985435Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9985877Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9986460Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9986926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9987421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9988047Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9988472Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9989051Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9989508Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9989942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:38.9990592Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9991341Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:38.9991916Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9992398Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:38.9992836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:38.9993303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:38.9993654Z dist init r=1, world=2 2022-11-23T03:30:38.9993905Z dist init r=0, world=2 2022-11-23T03:30:38.9994145Z ok (5.634s) 2022-11-23T03:30:38.9994619Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:38.9995284Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15549 2022-11-23T03:30:38.9995807Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15550 2022-11-23T03:30:38.9996421Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9996859Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9997437Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:38.9997893Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:38.9998332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:38.9998941Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:38.9999379Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:38.9999965Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0000425Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0000862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0001510Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0002189Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0002755Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0003240Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0003676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0004364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0004718Z dist init r=0, world=2 2022-11-23T03:30:39.0004970Z dist init r=1, world=2 2022-11-23T03:30:39.0005210Z ok (5.433s) 2022-11-23T03:30:39.0005680Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0006334Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15696 2022-11-23T03:30:39.0006849Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15697 2022-11-23T03:30:39.0007468Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0007955Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0008608Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0009071Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0009506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0010118Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0010556Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0011135Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0011588Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0012024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0012676Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0013358Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0013913Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0014397Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0014839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0015307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0015655Z dist init r=1, world=2 2022-11-23T03:30:39.0015904Z dist init r=0, world=2 2022-11-23T03:30:39.0016145Z ok (5.436s) 2022-11-23T03:30:39.0016625Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0017290Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15843 2022-11-23T03:30:39.0017805Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15844 2022-11-23T03:30:39.0018422Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0018859Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0019439Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0019899Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0020335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0020945Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0021458Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0022043Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0022503Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0022943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0023590Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0024270Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0024840Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0025375Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0025820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0026278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0026631Z dist init r=0, world=2 2022-11-23T03:30:39.0026884Z dist init r=1, world=2 2022-11-23T03:30:39.0027130Z ok (4.838s) 2022-11-23T03:30:39.0027602Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0028258Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15990 2022-11-23T03:30:39.0028773Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15991 2022-11-23T03:30:39.0029394Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0029834Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0030414Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0030873Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0031311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0031926Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0032368Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0032950Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0033406Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0033853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0034511Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0035188Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0035751Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0036234Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0036676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0037141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0037495Z dist init r=0, world=2 2022-11-23T03:30:39.0037748Z dist init r=1, world=2 2022-11-23T03:30:39.0037992Z ok (5.433s) 2022-11-23T03:30:39.0038542Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0039199Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16137 2022-11-23T03:30:39.0039713Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16138 2022-11-23T03:30:39.0040333Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0040774Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0041350Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0041808Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0042297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0042915Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0043354Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0043932Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0044391Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0044826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0045468Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0046151Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0046703Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0047203Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0047641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0048213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0048563Z dist init r=1, world=2 2022-11-23T03:30:39.0048813Z dist init r=0, world=2 2022-11-23T03:30:39.0049046Z ok (5.133s) 2022-11-23T03:30:39.0049537Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0050184Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16284 2022-11-23T03:30:39.0050709Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16285 2022-11-23T03:30:39.0051327Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0051764Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0052343Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0052801Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0053225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0053847Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0054291Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0054868Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0055412Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0055846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0056494Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0057174Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0057721Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0058221Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0058666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0059176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0059537Z dist init r=1, world=2 2022-11-23T03:30:39.0059792Z dist init r=0, world=2 2022-11-23T03:30:39.0060019Z ok (5.430s) 2022-11-23T03:30:39.0060506Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0061166Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16431 2022-11-23T03:30:39.0061681Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16432 2022-11-23T03:30:39.0062302Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0062743Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0063328Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0063796Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0064221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0064848Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0065290Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0065872Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0066332Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0066766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0067416Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0068099Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0068651Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0069148Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0069588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0070056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0070407Z dist init r=1, world=2 2022-11-23T03:30:39.0070659Z dist init r=0, world=2 2022-11-23T03:30:39.0070886Z ok (5.434s) 2022-11-23T03:30:39.0071375Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0072132Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16578 2022-11-23T03:30:39.0072649Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16579 2022-11-23T03:30:39.0073269Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0073708Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0074286Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0074744Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0075166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0075797Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0076291Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0076881Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0077338Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0077772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0078420Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0079085Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0079651Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0080154Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0080603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0081071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0081421Z dist init r=1, world=2 2022-11-23T03:30:39.0081674Z dist init r=0, world=2 2022-11-23T03:30:39.0081902Z ok (5.832s) 2022-11-23T03:30:39.0082386Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0083040Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16725 2022-11-23T03:30:39.0083556Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16726 2022-11-23T03:30:39.0084170Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0084619Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0085205Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0085668Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0086094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0086467Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0086634Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0087017Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0087199Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0087428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0087951Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0088349Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0088629Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0088903Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0089120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0089341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0089445Z dist init r=0, world=2 2022-11-23T03:30:39.0089551Z dist init r=1, world=2 2022-11-23T03:30:39.0089647Z ok (5.834s) 2022-11-23T03:30:39.0090061Z test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0090361Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16872 2022-11-23T03:30:39.0090570Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16873 2022-11-23T03:30:39.0090952Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0091107Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0091492Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0091676Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0091916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0092290Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0092458Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0092838Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0093019Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0093246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0093642Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0094035Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0094316Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0094592Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0094811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0095025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0095130Z dist init r=1, world=2 2022-11-23T03:30:39.0095234Z dist init r=0, world=2 2022-11-23T03:30:39.0095333Z ok (5.535s) 2022-11-23T03:30:39.0095665Z test_fsdp_state_dict_keys_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17019 2022-11-23T03:30:39.0095875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17020 2022-11-23T03:30:39.0096255Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0096479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0096866Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0097050Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0097280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0097652Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0097817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0098203Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0098387Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0098657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0099063Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0099451Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0099730Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0100005Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0100224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0100439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0100545Z dist init r=0, world=2 2022-11-23T03:30:39.0101624Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:30:39.0101733Z warnings.warn( 2022-11-23T03:30:39.0101839Z dist init r=1, world=2 2022-11-23T03:30:39.0102888Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:30:39.0102997Z warnings.warn( 2022-11-23T03:30:39.0103091Z ok (5.330s) 2022-11-23T03:30:39.0103426Z test_fsdp_state_dict_keys_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17162 2022-11-23T03:30:39.0103639Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17163 2022-11-23T03:30:39.0104013Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0104183Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0104551Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0104733Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0104960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0105333Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0105562Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0105949Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0106129Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0106355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0106749Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0107143Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0107420Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0107696Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0107961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0108188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0108295Z dist init r=0, world=2 2022-11-23T03:30:39.0109357Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:30:39.0109465Z warnings.warn( 2022-11-23T03:30:39.0109572Z dist init r=1, world=2 2022-11-23T03:30:39.0110621Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:30:39.0110732Z warnings.warn( 2022-11-23T03:30:39.0110827Z ok (5.429s) 2022-11-23T03:30:39.0111148Z test_fsdp_state_dict_keys_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17305 2022-11-23T03:30:39.0111357Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17306 2022-11-23T03:30:39.0111730Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0111900Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0112286Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0112454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0112681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0113053Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0113222Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0113606Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0113787Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0114010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0114406Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0114857Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0115134Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0115409Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0115625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0115841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0115947Z dist init r=0, world=2 2022-11-23T03:30:39.0117036Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:30:39.0117153Z warnings.warn( 2022-11-23T03:30:39.0117259Z dist init r=1, world=2 2022-11-23T03:30:39.0118305Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:30:39.0118410Z warnings.warn( 2022-11-23T03:30:39.0118507Z ok (5.230s) 2022-11-23T03:30:39.0118835Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0119302Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17448 2022-11-23T03:30:39.0119511Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17449 2022-11-23T03:30:39.0119885Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0120052Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0120436Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0120617Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0120832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0121213Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0121381Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0121763Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0121944Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0122172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0122567Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0122965Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0123243Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0123516Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0123790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0124005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0124113Z dist init r=0, world=2 2022-11-23T03:30:39.0124755Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:30:39.0124861Z warnings.warn( 2022-11-23T03:30:39.0124967Z dist init r=1, world=2 2022-11-23T03:30:39.0125599Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:30:39.0125705Z warnings.warn( 2022-11-23T03:30:39.0125803Z ok (5.232s) 2022-11-23T03:30:39.0126173Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0126637Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17595 2022-11-23T03:30:39.0126846Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17596 2022-11-23T03:30:39.0127220Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0127375Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0127866Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0128060Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0128296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0128675Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0128843Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0129228Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0129410Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0129636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0130034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0130425Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0130708Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0130986Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0131211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0131431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0131538Z dist init r=1, world=2 2022-11-23T03:30:39.0132175Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:30:39.0132283Z warnings.warn( 2022-11-23T03:30:39.0132388Z dist init r=0, world=2 2022-11-23T03:30:39.0133025Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:30:39.0133196Z warnings.warn( 2022-11-23T03:30:39.0133277Z ok (5.232s) 2022-11-23T03:30:39.0133589Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_rank0_only_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0134050Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17742 2022-11-23T03:30:39.0134262Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17743 2022-11-23T03:30:39.0134636Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0134804Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0135186Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0135419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0135653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0136030Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0136196Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0136580Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0136763Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0136988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0137385Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0137783Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0138062Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0138337Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0138553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0138774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0138879Z dist init r=1, world=2 2022-11-23T03:30:39.0139511Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:30:39.0139619Z warnings.warn( 2022-11-23T03:30:39.0139710Z dist init r=0, world=2 2022-11-23T03:30:39.0140346Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:30:39.0140451Z warnings.warn( 2022-11-23T03:30:39.0140545Z ok (5.431s) 2022-11-23T03:30:39.0140854Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_rank0_only_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0141310Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17889 2022-11-23T03:30:39.0141520Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17890 2022-11-23T03:30:39.0141890Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0142058Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0142502Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0142685Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0142909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0143281Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0143448Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0143831Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0144013Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0144239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0144701Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0145102Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0145381Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0145658Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0145873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0146090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0146180Z dist init r=0, world=2 2022-11-23T03:30:39.0146823Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:30:39.0146934Z warnings.warn( 2022-11-23T03:30:39.0147040Z dist init r=1, world=2 2022-11-23T03:30:39.0147671Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:30:39.0147776Z warnings.warn( 2022-11-23T03:30:39.0147871Z ok (5.133s) 2022-11-23T03:30:39.0148185Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0148642Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18036 2022-11-23T03:30:39.0148855Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18037 2022-11-23T03:30:39.0149233Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0149405Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0149789Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0149970Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0150196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0150567Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0150734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0151116Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0151298Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0151588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0151985Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0152376Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0152639Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0152915Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0153132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0153346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0153451Z dist init r=1, world=2 2022-11-23T03:30:39.0154121Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:30:39.0154231Z warnings.warn( 2022-11-23T03:30:39.0154334Z dist init r=0, world=2 2022-11-23T03:30:39.0154967Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:30:39.0155070Z warnings.warn( 2022-11-23T03:30:39.0155164Z ok (5.531s) 2022-11-23T03:30:39.0155473Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0155928Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18183 2022-11-23T03:30:39.0156143Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18184 2022-11-23T03:30:39.0156516Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0156684Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0157066Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0157247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0157471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0157839Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0158007Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0158393Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0158562Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0158790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0159184Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0159573Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0159852Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0160123Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0160338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0160557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0160716Z dist init r=0, world=2 2022-11-23T03:30:39.0161348Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:30:39.0161454Z warnings.warn( 2022-11-23T03:30:39.0161558Z dist init r=1, world=2 2022-11-23T03:30:39.0162185Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:30:39.0162288Z warnings.warn( 2022-11-23T03:30:39.0162382Z ok (5.632s) 2022-11-23T03:30:39.0162711Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0163217Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18330 2022-11-23T03:30:39.0163430Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18331 2022-11-23T03:30:39.0163802Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0163969Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0164352Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0164535Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0164746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0165121Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0165293Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0165673Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0165852Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0166075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0166470Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0166862Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0167136Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0167412Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0167637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0167910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0168016Z dist init r=1, world=2 2022-11-23T03:30:39.0168650Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:30:39.0168753Z warnings.warn( 2022-11-23T03:30:39.0168856Z dist init r=0, world=2 2022-11-23T03:30:39.0169484Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:30:39.0169587Z warnings.warn( 2022-11-23T03:30:39.0169680Z ok (5.433s) 2022-11-23T03:30:39.0170075Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0170537Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18477 2022-11-23T03:30:39.0170749Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18478 2022-11-23T03:30:39.0171122Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0171278Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0171659Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0171842Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0172118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0172499Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0172666Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0173049Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0173230Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0173456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0173852Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0174244Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0174524Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0174801Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0175019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0175233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0175340Z dist init r=1, world=2 2022-11-23T03:30:39.0175972Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:30:39.0176075Z warnings.warn( 2022-11-23T03:30:39.0176182Z dist init r=0, world=2 2022-11-23T03:30:39.0176820Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:30:39.0176925Z warnings.warn( 2022-11-23T03:30:39.0177005Z ok (5.333s) 2022-11-23T03:30:39.0177321Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_rank0_only_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0177779Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18624 2022-11-23T03:30:39.0177987Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18625 2022-11-23T03:30:39.0178360Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0178526Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0178911Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0179147Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0179373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0179745Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0179911Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0180295Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0180475Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0180700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0181095Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0181533Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0181817Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0182090Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0182315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0182532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0182636Z dist init r=0, world=2 2022-11-23T03:30:39.0183355Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:30:39.0183476Z warnings.warn( 2022-11-23T03:30:39.0183590Z dist init r=1, world=2 2022-11-23T03:30:39.0184359Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:30:39.0184483Z warnings.warn( 2022-11-23T03:30:39.0184599Z ok (5.433s) 2022-11-23T03:30:39.0184979Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_rank0_only_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0185533Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18771 2022-11-23T03:30:39.0185787Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18772 2022-11-23T03:30:39.0186237Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0186443Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0186912Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0187126Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0187397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0187844Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0188042Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0188501Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0188724Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0188993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0189555Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0190038Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0190368Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0190695Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0190954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0191196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0191326Z dist init r=0, world=2 2022-11-23T03:30:39.0192133Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:30:39.0192260Z warnings.warn( 2022-11-23T03:30:39.0192384Z dist init r=1, world=2 2022-11-23T03:30:39.0193096Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:30:39.0193198Z warnings.warn( 2022-11-23T03:30:39.0193292Z ok (5.632s) 2022-11-23T03:30:39.0193605Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0194061Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18918 2022-11-23T03:30:39.0194271Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18919 2022-11-23T03:30:39.0194649Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0194819Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0195205Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0195386Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0195612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0195982Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0196149Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0196530Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0196713Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0196940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0197335Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0197724Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0197989Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0198265Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0198483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0198699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0198808Z dist init r=1, world=2 2022-11-23T03:30:39.0198912Z dist init r=0, world=2 2022-11-23T03:30:39.0199075Z ok (5.332s) 2022-11-23T03:30:39.0199388Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0199853Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19065 2022-11-23T03:30:39.0200061Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19066 2022-11-23T03:30:39.0200435Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0200602Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0200987Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0201168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0201444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0201823Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0201992Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0202372Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0202551Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0202776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0203167Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0203545Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0203830Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0204105Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0204322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0204539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0204644Z dist init r=0, world=2 2022-11-23T03:30:39.0204749Z dist init r=1, world=2 2022-11-23T03:30:39.0204843Z ok (5.131s) 2022-11-23T03:30:39.0205142Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_rank0_only_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0205601Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19212 2022-11-23T03:30:39.0205815Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19213 2022-11-23T03:30:39.0206190Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0206360Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0206747Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0206926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0207153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0207525Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0207738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0208127Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0208379Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0208590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0208987Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0209380Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0209654Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0209933Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0210149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0210363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0210522Z dist init r=1, world=2 2022-11-23T03:30:39.0210629Z dist init r=0, world=2 2022-11-23T03:30:39.0210725Z ok (5.032s) 2022-11-23T03:30:39.0211021Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_rank0_only_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0211478Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19359 2022-11-23T03:30:39.0211683Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19360 2022-11-23T03:30:39.0212056Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0212221Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0212607Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0212795Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0213025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0213396Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0213563Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0213943Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0214108Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0214330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0214724Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0215119Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0215399Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0215674Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0215892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0216108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0216215Z dist init r=0, world=2 2022-11-23T03:30:39.0216319Z dist init r=1, world=2 2022-11-23T03:30:39.0216412Z ok (5.730s) 2022-11-23T03:30:39.0216711Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0217166Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19506 2022-11-23T03:30:39.0217472Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19507 2022-11-23T03:30:39.0217848Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0218015Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0218397Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0218575Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0218799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0219171Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0219337Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0219748Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0219940Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0220164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0220560Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0220956Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0221234Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0221508Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0221726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0221946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0222054Z dist init r=1, world=2 2022-11-23T03:30:39.0222158Z dist init r=0, world=2 2022-11-23T03:30:39.0222254Z ok (5.532s) 2022-11-23T03:30:39.0222554Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0223010Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19653 2022-11-23T03:30:39.0223220Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19654 2022-11-23T03:30:39.0223593Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0223762Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0224148Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0224333Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0224563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0224934Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0225086Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0225468Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0225651Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0225875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0226270Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0226728Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0227004Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0227280Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0227496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0227710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0227816Z dist init r=1, world=2 2022-11-23T03:30:39.0227923Z dist init r=0, world=2 2022-11-23T03:30:39.0228019Z ok (6.031s) 2022-11-23T03:30:39.0228338Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0228836Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19800 2022-11-23T03:30:39.0229052Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19801 2022-11-23T03:30:39.0229425Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0229595Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0229984Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0230165Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0230391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0230746Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0230917Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0231301Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0231480Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0231702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0232098Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0232491Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0232768Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0233040Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0233263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0233486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0233591Z dist init r=1, world=2 2022-11-23T03:30:39.0233696Z dist init r=0, world=2 2022-11-23T03:30:39.0233790Z ok (5.432s) 2022-11-23T03:30:39.0234107Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0234561Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19947 2022-11-23T03:30:39.0234768Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19948 2022-11-23T03:30:39.0235138Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0235306Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0235751Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0235935Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0236146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0236517Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0236684Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0237064Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0237247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0237473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0237914Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0238309Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0238587Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0238860Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0239074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0239289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0239397Z dist init r=0, world=2 2022-11-23T03:30:39.0239502Z dist init r=1, world=2 2022-11-23T03:30:39.0239596Z ok (5.643s) 2022-11-23T03:30:39.0239900Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_rank0_only_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0240355Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20094 2022-11-23T03:30:39.0240560Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20095 2022-11-23T03:30:39.0240929Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0241099Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0241464Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0241649Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0241874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0242247Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0242422Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0242803Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0242984Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0243209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0243603Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0243997Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0244273Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0244550Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0244828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0245050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0245154Z dist init r=0, world=2 2022-11-23T03:30:39.0245258Z dist init r=1, world=2 2022-11-23T03:30:39.0245352Z ok (5.332s) 2022-11-23T03:30:39.0245654Z test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_rank0_only_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0246115Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20241 2022-11-23T03:30:39.0246325Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20242 2022-11-23T03:30:39.0246698Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0246900Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0247293Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0247473Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0247807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0248188Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0248356Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0248739Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0248918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0249146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0249544Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0249932Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0250212Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0250486Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0250706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0250922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0251032Z dist init r=0, world=2 2022-11-23T03:30:39.0251135Z dist init r=1, world=2 2022-11-23T03:30:39.0251228Z ok (5.547s) 2022-11-23T03:30:39.0251541Z test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0251849Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20388 2022-11-23T03:30:39.0252058Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20389 2022-11-23T03:30:39.0252417Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0252584Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0252968Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0253150Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0253377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0253830Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0253996Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0254377Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0254557Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0254783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0255181Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0255572Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0255897Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0256179Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0256395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0256610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0256715Z dist init r=0, world=2 2022-11-23T03:30:39.0256819Z dist init r=1, world=2 2022-11-23T03:30:39.0256912Z ok (6.737s) 2022-11-23T03:30:39.0257221Z test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0257525Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20541 2022-11-23T03:30:39.0257718Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20542 2022-11-23T03:30:39.0258102Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0258271Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0258652Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0258831Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0259063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0259433Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0259598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0259979Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0260162Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0260391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0260784Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0261179Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0261456Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0261729Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0261947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0262163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0262271Z dist init r=0, world=2 2022-11-23T03:30:39.0262433Z dist init r=1, world=2 2022-11-23T03:30:39.0262532Z ok (5.232s) 2022-11-23T03:30:39.0262823Z test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0263129Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20684 2022-11-23T03:30:39.0263337Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20685 2022-11-23T03:30:39.0263719Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0263890Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0264271Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0264450Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0264726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0265100Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0265268Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0265651Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0265829Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0266056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0266452Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0266852Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0267137Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0267407Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0267622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0267835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0267942Z dist init r=0, world=2 2022-11-23T03:30:39.0268050Z dist init r=1, world=2 2022-11-23T03:30:39.0268129Z ok (6.935s) 2022-11-23T03:30:39.0268433Z test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0268735Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20837 2022-11-23T03:30:39.0268950Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20838 2022-11-23T03:30:39.0269326Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0269492Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0269876Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0270055Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0270281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0270650Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0270816Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0271199Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0271431Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0271657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0272061Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0272453Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0272731Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0273006Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0273226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0273438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0273589Z dist init r=0, world=2 2022-11-23T03:30:39.0273682Z dist init r=1, world=2 2022-11-23T03:30:39.0273780Z ok (5.030s) 2022-11-23T03:30:39.0274096Z test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0274398Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20980 2022-11-23T03:30:39.0274609Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20981 2022-11-23T03:30:39.0274983Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0275149Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0275529Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0275718Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0275944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0276314Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0276481Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0276861Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0277044Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0277268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0277662Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0278060Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0278339Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0278617Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0278837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0279052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0279156Z dist init r=0, world=2 2022-11-23T03:30:39.0279247Z dist init r=1, world=2 2022-11-23T03:30:39.0279342Z ok (7.036s) 2022-11-23T03:30:39.0279656Z test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0279961Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21133 2022-11-23T03:30:39.0280228Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21134 2022-11-23T03:30:39.0280611Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0280776Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0281161Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0281344Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0281571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0281942Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0282166Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0282557Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0282737Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0282962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0283355Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0283746Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0284022Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0284293Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0284508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0284731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0284822Z dist init r=0, world=2 2022-11-23T03:30:39.0284930Z dist init r=1, world=2 2022-11-23T03:30:39.0285023Z ok (4.831s) 2022-11-23T03:30:39.0285336Z test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0285636Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21276 2022-11-23T03:30:39.0285849Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21277 2022-11-23T03:30:39.0286221Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0286393Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0286785Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0286967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0287191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0287560Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0287774Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0288163Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0288342Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0288569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0288972Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0289445Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0289726Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0290002Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0290220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0290421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0290527Z dist init r=0, world=2 2022-11-23T03:30:39.0290632Z dist init r=1, world=2 2022-11-23T03:30:39.0290728Z ok (7.037s) 2022-11-23T03:30:39.0291037Z test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0291398Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21429 2022-11-23T03:30:39.0291611Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21430 2022-11-23T03:30:39.0291988Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0292155Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0292538Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0292722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0292951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0293324Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0293494Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0293877Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0294060Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0294284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0294679Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0295073Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0295348Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0295626Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0295837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0296056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0296162Z dist init r=0, world=2 2022-11-23T03:30:39.0296267Z dist init r=1, world=2 2022-11-23T03:30:39.0296362Z ok (4.732s) 2022-11-23T03:30:39.0296664Z test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0296967Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21572 2022-11-23T03:30:39.0297178Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21573 2022-11-23T03:30:39.0297559Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0297785Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0298174Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0298358Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0298587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0298960Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0299127Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0299509Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0299689Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0299962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0300366Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0300759Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0301038Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0301299Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0301519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0301741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0301848Z dist init r=0, world=2 2022-11-23T03:30:39.0301952Z dist init r=1, world=2 2022-11-23T03:30:39.0302048Z ok (6.834s) 2022-11-23T03:30:39.0302350Z test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0302658Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21725 2022-11-23T03:30:39.0302867Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21726 2022-11-23T03:30:39.0303248Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0303417Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0303801Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0303982Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0304211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0304588Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0304754Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0305138Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0305321Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0305545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0305940Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0306332Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0306596Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0306932Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0307148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0307363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0307474Z dist init r=0, world=2 2022-11-23T03:30:39.0307577Z dist init r=1, world=2 2022-11-23T03:30:39.0307671Z ok (6.936s) 2022-11-23T03:30:39.0307971Z test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0308274Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21878 2022-11-23T03:30:39.0308484Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21879 2022-11-23T03:30:39.0308908Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0309082Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0309468Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0309648Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0309876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0310248Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0310414Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0310795Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0310978Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0311210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0311606Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0311983Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0312261Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0312534Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0312749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0312968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0313076Z dist init r=0, world=2 2022-11-23T03:30:39.0313181Z dist init r=1, world=2 2022-11-23T03:30:39.0313281Z ok (7.035s) 2022-11-23T03:30:39.0313583Z test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0313883Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22031 2022-11-23T03:30:39.0314094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22032 2022-11-23T03:30:39.0314472Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0314640Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0315026Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0315210Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0315496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0315874Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0316039Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0316418Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0316600Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0316811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0317207Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0317599Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0317923Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0318202Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0318425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0318643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0318750Z dist init r=0, world=2 2022-11-23T03:30:39.0318853Z dist init r=1, world=2 2022-11-23T03:30:39.0318949Z ok (7.438s) 2022-11-23T03:30:39.0319247Z test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0319690Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22184 2022-11-23T03:30:39.0319902Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22185 2022-11-23T03:30:39.0320279Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0320448Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0320833Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0321014Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0321240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0321612Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0321778Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0322160Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0322332Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0322557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0322956Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0323348Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0323626Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0323903Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0324125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0324341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0324447Z dist init r=0, world=2 2022-11-23T03:30:39.0325147Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:30:39.0325253Z warnings.warn( 2022-11-23T03:30:39.0325360Z dist init r=1, world=2 2022-11-23T03:30:39.0325988Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:30:39.0326092Z warnings.warn( 2022-11-23T03:30:39.0326190Z ok (6.735s) 2022-11-23T03:30:39.0326484Z test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0327116Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22337 2022-11-23T03:30:39.0327334Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22338 2022-11-23T03:30:39.0327745Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0327915Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0328299Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0328480Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0328692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0329057Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0329216Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0329603Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0329778Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0329998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0330387Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0330770Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0331042Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0331313Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0331528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0331746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0331846Z dist init r=1, world=2 2022-11-23T03:30:39.0332889Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:30:39.0332990Z warnings.warn( 2022-11-23T03:30:39.0333616Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:30:39.0333715Z warnings.warn( 2022-11-23T03:30:39.0333893Z dist init r=0, world=2 2022-11-23T03:30:39.0334941Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:30:39.0335039Z warnings.warn( 2022-11-23T03:30:39.0335662Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:30:39.0335760Z warnings.warn( 2022-11-23T03:30:39.0335849Z ok (6.836s) 2022-11-23T03:30:39.0336190Z test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0336630Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22490 2022-11-23T03:30:39.0336836Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22491 2022-11-23T03:30:39.0337202Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0337362Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0337730Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0337906Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0338129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0338502Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0338666Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0339042Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0339217Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0339440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0339831Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0340217Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0340490Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0340766Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0340979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0341194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0341293Z dist init r=1, world=2 2022-11-23T03:30:39.0341392Z dist init r=0, world=2 2022-11-23T03:30:39.0341484Z ok (5.333s) 2022-11-23T03:30:39.0341772Z test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0342203Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22633 2022-11-23T03:30:39.0342407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22634 2022-11-23T03:30:39.0342776Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0342987Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0343365Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0343541Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0343852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0344234Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0344404Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0344793Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0344981Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0345266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0345680Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0346080Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0346366Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0346648Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0346872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0347098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0347208Z dist init r=1, world=2 2022-11-23T03:30:39.0347319Z dist init r=0, world=2 2022-11-23T03:30:39.0347427Z ok (5.031s) 2022-11-23T03:30:39.0347772Z test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False_fsdp_root_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0348220Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22776 2022-11-23T03:30:39.0348438Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22777 2022-11-23T03:30:39.0348798Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0348979Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0349375Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0349755Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0350001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0350403Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0350559Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0350949Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0351140Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0351377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0351787Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0352191Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0352512Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0352894Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0353123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0353352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0353467Z dist init r=0, world=2 2022-11-23T03:30:39.0353581Z dist init r=1, world=2 2022-11-23T03:30:39.0353684Z ok (6.639s) 2022-11-23T03:30:39.0353981Z test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0354469Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22929 2022-11-23T03:30:39.0354689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22930 2022-11-23T03:30:39.0376922Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0377137Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0377618Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0377807Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0378036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0378412Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0378567Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0378954Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0379138Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0379362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0379762Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0380158Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0380433Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0380705Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0380918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0381133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0381233Z dist init r=0, world=2 2022-11-23T03:30:39.0382297Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:30:39.0382399Z warnings.warn( 2022-11-23T03:30:39.0382499Z dist init r=1, world=2 2022-11-23T03:30:39.0383553Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:30:39.0383727Z warnings.warn( 2022-11-23T03:30:39.0383819Z ok (6.935s) 2022-11-23T03:30:39.0384094Z test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0384529Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23082 2022-11-23T03:30:39.0384732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23083 2022-11-23T03:30:39.0385100Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0385261Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0385640Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0385815Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0386091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0386467Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0386620Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0386999Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0387176Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0387397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0387793Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0388187Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0388465Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0388737Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0388952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0389163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0389264Z dist init r=0, world=2 2022-11-23T03:30:39.0389364Z dist init r=1, world=2 2022-11-23T03:30:39.0389453Z ok (7.931s) 2022-11-23T03:30:39.0389724Z test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0390160Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23235 2022-11-23T03:30:39.0390365Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23236 2022-11-23T03:30:39.0390744Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0390905Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0391286Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0391462Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0391683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0392041Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0392203Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0392583Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0392815Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0393045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0393443Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0393829Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0394105Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0394375Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0394589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0394801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0394950Z dist init r=1, world=2 2022-11-23T03:30:39.0395994Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:30:39.0396094Z warnings.warn( 2022-11-23T03:30:39.0396192Z dist init r=0, world=2 2022-11-23T03:30:39.0397230Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:30:39.0397334Z warnings.warn( 2022-11-23T03:30:39.0397423Z ok (6.837s) 2022-11-23T03:30:39.0397635Z test_state_dict_rank0_offload_save_load_flow_use_orig_params_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0397926Z Tests saving a model checkpoint only on rank 0 and loading it only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23388 2022-11-23T03:30:39.0398130Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23389 2022-11-23T03:30:39.0398501Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0398661Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0399041Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0399220Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0399444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0399805Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0399966Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0400344Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0400521Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0400741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0401132Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0401522Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0401857Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0402127Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0402338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0402548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0402649Z dist init r=1, world=2 2022-11-23T03:30:39.0402749Z dist init r=0, world=2 2022-11-23T03:30:39.0402838Z ok (5.631s) 2022-11-23T03:30:39.0403052Z test_state_dict_rank0_offload_save_load_flow_use_orig_params_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0403344Z Tests saving a model checkpoint only on rank 0 and loading it only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23535 2022-11-23T03:30:39.0403598Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23536 2022-11-23T03:30:39.0403983Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0404150Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0404534Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0404701Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0404925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0405291Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0405453Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0405841Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0406022Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0406244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0406642Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0407031Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0407309Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0407581Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0407849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0408071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0408179Z dist init r=1, world=2 2022-11-23T03:30:39.0408281Z dist init r=0, world=2 2022-11-23T03:30:39.0408372Z ok (5.131s) 2022-11-23T03:30:39.0408705Z test_state_dict_save_load_flow_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23682 2022-11-23T03:30:39.0408911Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23683 2022-11-23T03:30:39.0409284Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0409450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0409834Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0410003Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0410227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0410678Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0410844Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0411226Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0411401Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0411628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0412021Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0412411Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0412918Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0413203Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0413422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0413642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0414680Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:30:39.0414783Z warnings.warn( 2022-11-23T03:30:39.0415004Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:30:39.0416060Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:30:39.0416164Z warnings.warn( 2022-11-23T03:30:39.0416383Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:30:39.0416601Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:30:39.0416817Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:30:39.0416922Z dist init r=0, world=2 2022-11-23T03:30:39.0417027Z dist init r=1, world=2 2022-11-23T03:30:39.0417126Z ok (7.338s) 2022-11-23T03:30:39.0417459Z test_state_dict_save_load_flow_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23835 2022-11-23T03:30:39.0417667Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23836 2022-11-23T03:30:39.0418031Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0418198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0418583Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0418761Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0418986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0419363Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0419587Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0419976Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0420158Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0420384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0420778Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0421166Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0421443Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0421782Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0422004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0422217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0423253Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:30:39.0423356Z warnings.warn( 2022-11-23T03:30:39.0423573Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:30:39.0424616Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:30:39.0424722Z warnings.warn( 2022-11-23T03:30:39.0424940Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:30:39.0425159Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:30:39.0425372Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:30:39.0425476Z dist init r=0, world=2 2022-11-23T03:30:39.0425581Z dist init r=1, world=2 2022-11-23T03:30:39.0425662Z ok (7.534s) 2022-11-23T03:30:39.0425988Z test_state_dict_save_load_flow_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23988 2022-11-23T03:30:39.0426200Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23989 2022-11-23T03:30:39.0426578Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0426744Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0427125Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0427303Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0427529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0427899Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0428116Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0428499Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0428678Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0428905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0429301Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0429689Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0429965Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0430241Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0430497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0430719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0431767Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:30:39.0431875Z warnings.warn( 2022-11-23T03:30:39.0432096Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:30:39.0433129Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:30:39.0433235Z warnings.warn( 2022-11-23T03:30:39.0433450Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:30:39.0433672Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:30:39.0433886Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:30:39.0433978Z dist init r=0, world=2 2022-11-23T03:30:39.0434083Z dist init r=1, world=2 2022-11-23T03:30:39.0434178Z ok (6.833s) 2022-11-23T03:30:39.0434531Z test_state_dict_skip_module_state_dict_type_local_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24141 2022-11-23T03:30:39.0434740Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24142 2022-11-23T03:30:39.0435116Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0435282Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0435662Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0435845Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0436068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0436442Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0436609Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0437052Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0437232Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0437455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0437853Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0438245Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0438524Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0438798Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0439017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0439277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0439376Z dist init r=0, world=2 2022-11-23T03:30:39.0439482Z dist init r=1, world=2 2022-11-23T03:30:39.0439573Z ok (7.037s) 2022-11-23T03:30:39.0439924Z test_state_dict_skip_module_state_dict_type_sharded_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24294 2022-11-23T03:30:39.0440136Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24295 2022-11-23T03:30:39.0440514Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0440679Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0441066Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0441243Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0441476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0441845Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0442011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0442395Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0442574Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0442799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0443193Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0443586Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0443866Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0444141Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0444355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0444571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0444664Z dist init r=1, world=2 2022-11-23T03:30:39.0444768Z dist init r=0, world=2 2022-11-23T03:30:39.0445403Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:30:39.0445504Z warnings.warn( 2022-11-23T03:30:39.0446134Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:30:39.0446293Z warnings.warn( 2022-11-23T03:30:39.0446385Z ok (6.836s) 2022-11-23T03:30:39.0446725Z test_state_dict_skip_module_state_dict_type_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24447 2022-11-23T03:30:39.0446933Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24448 2022-11-23T03:30:39.0447313Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0447477Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0447901Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0448082Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0448374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0448756Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0448919Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0449300Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0449479Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0449704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0450100Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0450493Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0450774Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0451036Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0451253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0451467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0451574Z dist init r=0, world=2 2022-11-23T03:30:39.0451682Z dist init r=1, world=2 2022-11-23T03:30:39.0451775Z ok (6.735s) 2022-11-23T03:30:39.0452056Z test_state_dict_type (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24600 2022-11-23T03:30:39.0452264Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24601 2022-11-23T03:30:39.0452639Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0452805Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0453188Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0453366Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0453591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0453960Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0454126Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0454508Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0454687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0454977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0455380Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0455771Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0456037Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0456313Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0456531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0456744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0456847Z dist init r=1, world=2 2022-11-23T03:30:39.0456951Z dist init r=0, world=2 2022-11-23T03:30:39.0457045Z ok (4.826s) 2022-11-23T03:30:39.0457472Z test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24743 2022-11-23T03:30:39.0457688Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24744 2022-11-23T03:30:39.0458065Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0458232Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0458616Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0458795Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0459018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0459389Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0459557Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0459939Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0460119Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0460338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0460731Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0461119Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0461384Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0461659Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0461879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0462089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0462191Z dist init r=1, world=2 2022-11-23T03:30:39.0463225Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:30:39.0463324Z warnings.warn( 2022-11-23T03:30:39.0463954Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:30:39.0464109Z warnings.warn( 2022-11-23T03:30:39.0464214Z dist init r=0, world=2 2022-11-23T03:30:39.0465260Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:30:39.0465360Z warnings.warn( 2022-11-23T03:30:39.0465989Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:30:39.0466093Z warnings.warn( 2022-11-23T03:30:39.0466227Z ok (5.334s) 2022-11-23T03:30:39.0466611Z test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24890 2022-11-23T03:30:39.0466817Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24891 2022-11-23T03:30:39.0467188Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0467354Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0467734Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0467908Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0468133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0468507Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0468675Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0469058Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0469240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0469465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0469861Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0470240Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0470519Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0470796Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0471012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0471222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0471329Z dist init r=1, world=2 2022-11-23T03:30:39.0472078Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:386: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True) 2022-11-23T03:30:39.0472178Z warnings.warn( 2022-11-23T03:30:39.0472281Z dist init r=0, world=2 2022-11-23T03:30:39.0473007Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:386: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True) 2022-11-23T03:30:39.0473161Z warnings.warn( 2022-11-23T03:30:39.0473251Z ok (5.232s) 2022-11-23T03:30:39.0473636Z test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25037 2022-11-23T03:30:39.0473844Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25038 2022-11-23T03:30:39.0474217Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0474387Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0474774Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0474951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0475225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0475602Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0475764Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0476144Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0476322Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0476535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0476929Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0477324Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0477601Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0477875Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0478087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0478298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0478402Z dist init r=1, world=2 2022-11-23T03:30:39.0479434Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:30:39.0479540Z warnings.warn( 2022-11-23T03:30:39.0480166Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:30:39.0480266Z warnings.warn( 2022-11-23T03:30:39.0480366Z dist init r=0, world=2 2022-11-23T03:30:39.0481403Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:30:39.0481506Z warnings.warn( 2022-11-23T03:30:39.0482135Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:30:39.0482292Z warnings.warn( 2022-11-23T03:30:39.0482384Z ok (5.231s) 2022-11-23T03:30:39.0482766Z test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25184 2022-11-23T03:30:39.0482971Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25185 2022-11-23T03:30:39.0483344Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0483507Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0483888Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0484109Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0484336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0484708Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0484871Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0485251Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0485428Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0485641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0486033Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0486426Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0486701Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0486977Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0487190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0487406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0487511Z dist init r=0, world=2 2022-11-23T03:30:39.0488353Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:386: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True) 2022-11-23T03:30:39.0488459Z warnings.warn( 2022-11-23T03:30:39.0488566Z dist init r=1, world=2 2022-11-23T03:30:39.0489302Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:386: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True) 2022-11-23T03:30:39.0489404Z warnings.warn( 2022-11-23T03:30:39.0489496Z ok (5.132s) 2022-11-23T03:30:39.0489871Z test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25331 2022-11-23T03:30:39.0490082Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25332 2022-11-23T03:30:39.0490454Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0490621Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0491104Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0491282Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0491508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0491876Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0492041Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0492410Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0492589Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0492819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0493264Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0493666Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0493946Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0494219Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0494435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0494697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0494827Z dist init r=1, world=2 2022-11-23T03:30:39.0496100Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:30:39.0496229Z warnings.warn( 2022-11-23T03:30:39.0496349Z dist init r=0, world=2 2022-11-23T03:30:39.0497606Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:30:39.0497726Z warnings.warn( 2022-11-23T03:30:39.0497844Z ok (5.333s) 2022-11-23T03:30:39.0498289Z test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25478 2022-11-23T03:30:39.0498542Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25479 2022-11-23T03:30:39.0498999Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0499196Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0499654Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0499864Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0500132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0500590Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0500855Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0501319Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0501533Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0501790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0502268Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0502740Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0503072Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0503408Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0503718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0503981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0504101Z dist init r=1, world=2 2022-11-23T03:30:39.0504987Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:386: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True) 2022-11-23T03:30:39.0505109Z warnings.warn( 2022-11-23T03:30:39.0505238Z dist init r=0, world=2 2022-11-23T03:30:39.0506114Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:386: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True) 2022-11-23T03:30:39.0506233Z warnings.warn( 2022-11-23T03:30:39.0506349Z ok (5.034s) 2022-11-23T03:30:39.0506798Z test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25625 2022-11-23T03:30:39.0507053Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25626 2022-11-23T03:30:39.0507501Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0507702Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0508166Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0508380Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0508649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0509135Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0509587Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0509779Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0510238Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0510451Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0510719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0511199Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0511523Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0511863Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0512187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0512443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0512567Z dist init r=0, world=2 2022-11-23T03:30:39.0513746Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:30:39.0513852Z warnings.warn( 2022-11-23T03:30:39.0513956Z dist init r=1, world=2 2022-11-23T03:30:39.0515024Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:30:39.0515131Z warnings.warn( 2022-11-23T03:30:39.0515223Z ok (4.832s) 2022-11-23T03:30:39.0515589Z test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25772 2022-11-23T03:30:39.0515800Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25773 2022-11-23T03:30:39.0516174Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0516344Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0516729Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0516906Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0517130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0517502Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0517665Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0518051Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0518231Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0518443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0518843Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0519234Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0519513Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0519784Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0520002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0520215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0520319Z dist init r=1, world=2 2022-11-23T03:30:39.0521050Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:386: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True) 2022-11-23T03:30:39.0521207Z warnings.warn( 2022-11-23T03:30:39.0521308Z dist init r=0, world=2 2022-11-23T03:30:39.0522031Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:386: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True) 2022-11-23T03:30:39.0522132Z warnings.warn( 2022-11-23T03:30:39.0522226Z ok (5.229s) 2022-11-23T03:30:39.0522496Z test_state_dict_with_manual_ac_wrapper_state_dict_type_sharded_state_dict_rank0_only_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0522794Z Tests saving and loading a state dict for a model manually wrapped with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25919 2022-11-23T03:30:39.0523041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25920 2022-11-23T03:30:39.0523423Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0523590Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0523978Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0524158Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0524385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0524755Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0524908Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0525293Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0525479Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0525704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0526098Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0526485Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0526761Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0527035Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0527252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0527465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0527574Z dist init r=1, world=2 2022-11-23T03:30:39.0527676Z dist init r=0, world=2 2022-11-23T03:30:39.0527840Z ok (5.434s) 2022-11-23T03:30:39.0528105Z test_state_dict_with_manual_ac_wrapper_state_dict_type_sharded_state_dict_rank0_only_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0528407Z Tests saving and loading a state dict for a model manually wrapped with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26066 2022-11-23T03:30:39.0528612Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26067 2022-11-23T03:30:39.0528985Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0529149Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0529527Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0529772Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0529986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0530359Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0530524Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0530905Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0531083Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0531309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0531707Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0532143Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0532429Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0532702Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0532919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0533131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0533233Z dist init r=1, world=2 2022-11-23T03:30:39.0533333Z dist init r=0, world=2 2022-11-23T03:30:39.0533426Z ok (5.131s) 2022-11-23T03:30:39.0533684Z test_state_dict_with_manual_ac_wrapper_state_dict_type_state_dict_rank0_only_and_offload_False (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0533987Z Tests saving and loading a state dict for a model manually wrapped with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26209 2022-11-23T03:30:39.0534195Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26210 2022-11-23T03:30:39.0534570Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0534736Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0535119Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0535286Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0535510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0535878Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0536043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0536425Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0536606Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0536829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0537222Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0537606Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0537881Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0538154Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0538368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0538582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0538741Z dist init r=0, world=2 2022-11-23T03:30:39.0538841Z dist init r=1, world=2 2022-11-23T03:30:39.0538931Z ok (5.534s) 2022-11-23T03:30:39.0539182Z test_state_dict_with_manual_ac_wrapper_state_dict_type_state_dict_rank0_only_and_offload_True (__main__.TestFSDPStateDict) 2022-11-23T03:30:39.0539478Z Tests saving and loading a state dict for a model manually wrapped with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26356 2022-11-23T03:30:39.0539683Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26357 2022-11-23T03:30:39.0540060Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0540213Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0540593Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0540819Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0541047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0541421Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0541585Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0541965Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0542141Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0542367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0542761Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0543154Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0543429Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0543701Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0543911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0544123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0544224Z dist init r=1, world=2 2022-11-23T03:30:39.0544326Z dist init r=0, world=2 2022-11-23T03:30:39.0544417Z ok (5.330s) 2022-11-23T03:30:39.0544760Z test_state_dict_with_shared_parameters_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26503 2022-11-23T03:30:39.0544968Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26504 2022-11-23T03:30:39.0545344Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0545496Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0545875Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0546054Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0546274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0546642Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0546806Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0547182Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0547412Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0547637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0548033Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0548421Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0548698Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0548967Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0549176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0549385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0549483Z dist init r=0, world=2 2022-11-23T03:30:39.0549628Z dist init r=1, world=2 2022-11-23T03:30:39.0549719Z ok (5.030s) 2022-11-23T03:30:39.0550063Z test_state_dict_with_shared_parameters_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26646 2022-11-23T03:30:39.0550264Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26647 2022-11-23T03:30:39.0550628Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0550788Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0551167Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0551341Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0551560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0551932Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0552093Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0552471Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0552645Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0552866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0553263Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0553653Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0553927Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0554202Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0554410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0554619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0554716Z dist init r=1, world=2 2022-11-23T03:30:39.0554815Z dist init r=0, world=2 2022-11-23T03:30:39.0554907Z ok (5.429s) 2022-11-23T03:30:39.0555238Z test_state_dict_with_shared_parameters_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26789 2022-11-23T03:30:39.0555442Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26790 2022-11-23T03:30:39.0555803Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0555966Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0556400Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0556574Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0556793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0557162Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0557322Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0557703Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0557876Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0558095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0558544Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0558935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0559209Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0559477Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0559687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0559897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0559997Z dist init r=1, world=2 2022-11-23T03:30:39.0560095Z dist init r=0, world=2 2022-11-23T03:30:39.0560183Z ok (5.028s) 2022-11-23T03:30:39.0560474Z test_wrong_state_dict_config (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26932 2022-11-23T03:30:39.0560670Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26933 2022-11-23T03:30:39.0561042Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0561209Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0561588Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0561764Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0561985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:39.0562353Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:39.0562517Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:39.0562902Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:39.0563077Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:39.0563302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:39.0563700Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0564093Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:39.0564373Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0564645Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:39.0564854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:39.0565123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:39.0565227Z dist init r=0, world=2 2022-11-23T03:30:39.0566261Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:30:39.0566362Z warnings.warn( 2022-11-23T03:30:39.0566466Z dist init r=1, world=2 2022-11-23T03:30:39.0567549Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:30:39.0567658Z warnings.warn( 2022-11-23T03:30:39.0567879Z ok (5.029s) 2022-11-23T03:30:39.0567888Z 2022-11-23T03:30:39.0568180Z ---------------------------------------------------------------------- 2022-11-23T03:30:39.0568292Z Ran 116 tests in 642.173s 2022-11-23T03:30:39.0568299Z 2022-11-23T03:30:39.0568372Z OK 2022-11-23T03:30:39.0568393Z 2022-11-23T03:30:39.0568497Z Generating XML reports... 2022-11-23T03:30:39.0568941Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict/TEST-TestFSDPStateDict-20221123031954.xml 2022-11-23T03:30:39.0568948Z 2022-11-23T03:30:39.0569475Z ##[endgroup] 2022-11-23T03:30:39.0569956Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_state_dict (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_state_dict_e84dnls9) 2022-11-23T03:30:39.0569971Z 2022-11-23T03:30:39.0570244Z Running distributed/fsdp/test_fsdp_pure_fp16 ... [2022-11-23 03:30:38.954568] 2022-11-23T03:30:39.0570733Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_pure_fp16.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:30:38.955238] 2022-11-23T03:30:50.0070820Z 2022-11-23T03:30:50.0071891Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_pure_fp16 2022-11-23T03:30:50.0075596Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_pure_fp16 (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_pure_fp16_dxq_prh9) 2022-11-23T03:30:50.0079006Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_pure_fp16 2022-11-23T03:30:50.0079733Z 2022-11-23T03:30:50.0080113Z Running tests... 2022-11-23T03:30:50.0081251Z ---------------------------------------------------------------------- 2022-11-23T03:30:50.0082379Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) 2022-11-23T03:30:50.0085645Z Tests pure FP16 training, including when the parameter's dtype is ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/73315 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.600s) 2022-11-23T03:30:50.0088085Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=True) (__main__.TestPureFP16) 2022-11-23T03:30:50.0089989Z Tests pure FP16 training, including when the parameter's dtype is ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27142 2022-11-23T03:30:50.0091383Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27143 2022-11-23T03:30:50.0093032Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:50.0094751Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:50.0096320Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:50.0097517Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:50.0098649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:30:50.0100294Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:30:50.0101434Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:30:50.0102946Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:30:50.0104143Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:30:50.0105483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:30:50.0107255Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:50.0109083Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:30:50.0110576Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:50.0111886Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:30:50.0113026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:30:50.0114233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:30:50.0115138Z dist init r=1, world=2 2022-11-23T03:30:50.0116342Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:30:50.0120102Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:30:50.0122147Z warnings.warn( 2022-11-23T03:30:50.0122812Z File "", line 1, in 2022-11-23T03:30:50.0123726Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:30:50.0124679Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:30:50.0125614Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:30:50.0126557Z return self._bootstrap(parent_sentinel) 2022-11-23T03:30:50.0127558Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:30:50.0128519Z self.run() 2022-11-23T03:30:50.0129355Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:30:50.0130296Z self._target(*self._args, **self._kwargs) 2022-11-23T03:30:50.0131703Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:30:50.0132704Z self.run_test(test_name, pipe) 2022-11-23T03:30:50.0134136Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:30:50.0135150Z getattr(self, test_name)() 2022-11-23T03:30:50.0136532Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:30:50.0137457Z fn() 2022-11-23T03:30:50.0138791Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:30:50.0140003Z test(self, **param_kwargs) 2022-11-23T03:30:50.0141402Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:30:50.0142401Z return func(*args, **kwargs) 2022-11-23T03:30:50.0143398Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_pure_fp16.py", line 47, in test_pure_fp16 2022-11-23T03:30:50.0144321Z self._test_fsdp_parity( 2022-11-23T03:30:50.0145733Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:30:50.0146812Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:30:50.0148306Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:30:50.0149319Z output = model(*input) 2022-11-23T03:30:50.0150753Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:30:50.0151774Z return forward_call(*input, **kwargs) 2022-11-23T03:30:50.0153242Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:30:50.0154407Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:30:50.0155924Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:30:50.0156930Z _lazy_init(state, module) 2022-11-23T03:30:50.0158301Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:30:50.0159339Z handle.init_flat_param_attributes() 2022-11-23T03:30:50.0160698Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:30:50.0161666Z return func(*args, **kwargs) 2022-11-23T03:30:50.0163118Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:30:50.0164115Z p_assert( 2022-11-23T03:30:50.0165371Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:30:50.0166335Z traceback.print_stack() 2022-11-23T03:30:50.0166999Z dist init r=0, world=2 2022-11-23T03:30:50.0168280Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:30:50.0172054Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:30:50.0174092Z warnings.warn( 2022-11-23T03:30:50.0174762Z File "", line 1, in 2022-11-23T03:30:50.0175698Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:30:50.0176646Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:30:50.0177581Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:30:50.0178524Z return self._bootstrap(parent_sentinel) 2022-11-23T03:30:50.0179510Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:30:50.0180346Z self.run() 2022-11-23T03:30:50.0181187Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:30:50.0182117Z self._target(*self._args, **self._kwargs) 2022-11-23T03:30:50.0183478Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:30:50.0184653Z self.run_test(test_name, pipe) 2022-11-23T03:30:50.0186064Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:30:50.0187050Z getattr(self, test_name)() 2022-11-23T03:30:50.0188425Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:30:50.0189356Z fn() 2022-11-23T03:30:50.0190674Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:30:50.0191689Z test(self, **param_kwargs) 2022-11-23T03:30:50.0193052Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:30:50.0194033Z return func(*args, **kwargs) 2022-11-23T03:30:50.0195042Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_pure_fp16.py", line 47, in test_pure_fp16 2022-11-23T03:30:50.0195983Z self._test_fsdp_parity( 2022-11-23T03:30:50.0197713Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:30:50.0198796Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:30:50.0200290Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:30:50.0201288Z output = model(*input) 2022-11-23T03:30:50.0202553Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:30:50.0203540Z return forward_call(*input, **kwargs) 2022-11-23T03:30:50.0205006Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:30:50.0206157Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:30:50.0207676Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:30:50.0208984Z _lazy_init(state, module) 2022-11-23T03:30:50.0210359Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:30:50.0211371Z handle.init_flat_param_attributes() 2022-11-23T03:30:50.0212727Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:30:50.0213703Z return func(*args, **kwargs) 2022-11-23T03:30:50.0215141Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:30:50.0216104Z p_assert( 2022-11-23T03:30:50.0217364Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:30:50.0218330Z traceback.print_stack() 2022-11-23T03:30:50.0218990Z ok (6.444s) 2022-11-23T03:30:50.0219347Z 2022-11-23T03:30:50.0220089Z ---------------------------------------------------------------------- 2022-11-23T03:30:50.0220936Z Ran 2 tests in 7.045s 2022-11-23T03:30:50.0221333Z 2022-11-23T03:30:50.0221576Z OK (skipped=1) 2022-11-23T03:30:50.0221820Z 2022-11-23T03:30:50.0221958Z Generating XML reports... 2022-11-23T03:30:50.0222636Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_pure_fp16/TEST-TestPureFP16-20221123033040.xml 2022-11-23T03:30:50.0222953Z 2022-11-23T03:30:50.0223230Z ##[endgroup] 2022-11-23T03:30:50.0223831Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_pure_fp16 (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_pure_fp16_dxq_prh9) 2022-11-23T03:30:50.0224166Z 2022-11-23T03:30:50.0224443Z Running distributed/fsdp/test_fsdp_optim_state ... [2022-11-23 03:30:50.007626] 2022-11-23T03:30:50.0225147Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_optim_state.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:30:50.008302] 2022-11-23T03:36:56.8281137Z 2022-11-23T03:36:56.8285103Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_optim_state 2022-11-23T03:36:56.8286632Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_optim_state (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_optim_state_yxx606yx) 2022-11-23T03:36:56.8288713Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state 2022-11-23T03:36:56.8289128Z 2022-11-23T03:36:56.8289303Z Running tests... 2022-11-23T03:36:56.8290105Z ---------------------------------------------------------------------- 2022-11-23T03:36:56.8299014Z test_flatten_sharded_optim_state_dict_nested (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8301451Z Tests :meth:`flatten_sharded_optim_state_dict` for an FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27362 2022-11-23T03:36:56.8303566Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27363 2022-11-23T03:36:56.8306216Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8307837Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8309863Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8311481Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8313087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8315586Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8316885Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8318504Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8319815Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8321038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8323275Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8325573Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8328084Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8330001Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8331658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8333503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8334762Z dist init r=1, world=2 2022-11-23T03:36:56.8335803Z dist init r=0, world=2 2022-11-23T03:36:56.8338943Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:36:56.8341252Z warnings.warn( 2022-11-23T03:36:56.8344348Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8346439Z warnings.warn( 2022-11-23T03:36:56.8349597Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:36:56.8351595Z warnings.warn( 2022-11-23T03:36:56.8354713Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8357318Z warnings.warn( 2022-11-23T03:36:56.8358238Z ok (7.427s) 2022-11-23T03:36:56.8359549Z test_flatten_sharded_optim_state_dict_transformer (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8362501Z Tests :meth:`flatten_sharded_optim_state_dict` for an FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27515 2022-11-23T03:36:56.8364631Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27516 2022-11-23T03:36:56.8367100Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8369192Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8371778Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8373678Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8375289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8376563Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8377335Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8378308Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8379067Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8379799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8380644Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8381488Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8382211Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8382848Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8383426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8383999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8384458Z dist init r=1, world=2 2022-11-23T03:36:56.8385430Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:36:56.8386117Z warnings.warn( 2022-11-23T03:36:56.8386418Z dist init r=0, world=2 2022-11-23T03:36:56.8387392Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:36:56.8388054Z warnings.warn( 2022-11-23T03:36:56.8388368Z ok (7.734s) 2022-11-23T03:36:56.8388765Z test_full_optim_state_dict_keys (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8389374Z Tests that the parameter keys returned by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27668 2022-11-23T03:36:56.8389974Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27669 2022-11-23T03:36:56.8390763Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8391322Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8392163Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8392748Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8393307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8394091Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8394652Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8395358Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8395937Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8396502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8397387Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8398266Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8398977Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8399614Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8400152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8400741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8401189Z dist init r=0, world=2 2022-11-23T03:36:56.8401515Z dist init r=1, world=2 2022-11-23T03:36:56.8403097Z ok (6.734s) 2022-11-23T03:36:56.8403517Z test_full_optim_state_dict_nested_invalid (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8404165Z Tests that :meth:`full_optim_state_dict` raises an error when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27821 2022-11-23T03:36:56.8404796Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27822 2022-11-23T03:36:56.8405572Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8406131Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8406858Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8407443Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8408103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8408903Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8409436Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8410171Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8410754Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8411299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8412129Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8412986Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8413707Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8414345Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8414888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8415567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8416016Z dist init r=1, world=2 2022-11-23T03:36:56.8416348Z dist init r=0, world=2 2022-11-23T03:36:56.8416666Z ok (6.832s) 2022-11-23T03:36:56.8417051Z test_optim_input_warning (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8417643Z Tests that passing the ``optim_input`` argument into optimizer state ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27974 2022-11-23T03:36:56.8418307Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27975 2022-11-23T03:36:56.8419085Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8419646Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8420435Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8421030Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8421586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8422352Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8422913Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8423633Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8424212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8424765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8425577Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8426332Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8426927Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8427427Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8427891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8428374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8428751Z dist init r=0, world=2 2022-11-23T03:36:56.8429031Z dist init r=1, world=2 2022-11-23T03:36:56.8429295Z ok (6.731s) 2022-11-23T03:36:56.8429768Z test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8430451Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28127 2022-11-23T03:36:56.8431008Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28128 2022-11-23T03:36:56.8431651Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8432118Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8432728Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8433212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8433674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8434303Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8434839Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8435450Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8435938Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8436405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8437083Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8437796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8438395Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8438891Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8439403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8439899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8440267Z dist init r=0, world=2 2022-11-23T03:36:56.8440544Z dist init r=1, world=2 2022-11-23T03:36:56.8441397Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8441979Z warnings.warn( 2022-11-23T03:36:56.8442792Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8443368Z warnings.warn( 2022-11-23T03:36:56.8443631Z ok (6.932s) 2022-11-23T03:36:56.8444133Z test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8444815Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28280 2022-11-23T03:36:56.8445370Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28281 2022-11-23T03:36:56.8446010Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8446471Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8447053Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8447540Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8448079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8448822Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8449377Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8450099Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8450675Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8451197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8452010Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8452867Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8453587Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8454312Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8454872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8455457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8455882Z dist init r=1, world=2 2022-11-23T03:36:56.8456205Z dist init r=0, world=2 2022-11-23T03:36:56.8457220Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8457914Z warnings.warn( 2022-11-23T03:36:56.8458987Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8459680Z warnings.warn( 2022-11-23T03:36:56.8459994Z ok (6.836s) 2022-11-23T03:36:56.8460590Z test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8461377Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28433 2022-11-23T03:36:56.8462036Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28434 2022-11-23T03:36:56.8462821Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8463383Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8464109Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8464687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8465238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8466030Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8466488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8467095Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8467582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8468047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8468736Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8469450Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8470047Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8470550Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8471014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8471500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8471871Z dist init r=0, world=2 2022-11-23T03:36:56.8472722Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8473293Z warnings.warn( 2022-11-23T03:36:56.8473636Z dist init r=1, world=2 2022-11-23T03:36:56.8474479Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8475021Z warnings.warn( 2022-11-23T03:36:56.8475282Z ok (6.832s) 2022-11-23T03:36:56.8475774Z test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8476447Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28586 2022-11-23T03:36:56.8476999Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28587 2022-11-23T03:36:56.8477712Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8478195Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8478775Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8479258Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8479723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8480382Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8480846Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8481460Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8481940Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8482408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8483060Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8483784Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8484378Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8484906Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8485375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8485861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8486242Z dist init r=1, world=2 2022-11-23T03:36:56.8487063Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8487630Z warnings.warn( 2022-11-23T03:36:56.8488253Z dist init r=0, world=2 2022-11-23T03:36:56.8489340Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8490024Z warnings.warn( 2022-11-23T03:36:56.8490344Z ok (6.932s) 2022-11-23T03:36:56.8490936Z test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8491766Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28739 2022-11-23T03:36:56.8492501Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28740 2022-11-23T03:36:56.8493279Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8493838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8494565Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8495143Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8495692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8496485Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8497055Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8497833Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8498419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8498970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8499784Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8500638Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8501357Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8501988Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8502521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8503116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8503566Z dist init r=0, world=2 2022-11-23T03:36:56.8503901Z dist init r=1, world=2 2022-11-23T03:36:56.8504913Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8505596Z warnings.warn( 2022-11-23T03:36:56.8506508Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8507080Z warnings.warn( 2022-11-23T03:36:56.8507315Z ok (6.831s) 2022-11-23T03:36:56.8507807Z test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8508486Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28892 2022-11-23T03:36:56.8509041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28893 2022-11-23T03:36:56.8509680Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8510150Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8510754Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8511213Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8511672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8512395Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8512856Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8513460Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8513945Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8514414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8515098Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8515778Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8516366Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8516944Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8517413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8517892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8518273Z dist init r=1, world=2 2022-11-23T03:36:56.8519120Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8519666Z warnings.warn( 2022-11-23T03:36:56.8519937Z dist init r=0, world=2 2022-11-23T03:36:56.8520777Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8521349Z warnings.warn( 2022-11-23T03:36:56.8521610Z ok (6.831s) 2022-11-23T03:36:56.8522100Z test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8522776Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29045 2022-11-23T03:36:56.8523327Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29046 2022-11-23T03:36:56.8523942Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8524408Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8525017Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8525512Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8525981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8526631Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8527094Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8527670Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8528222Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8528753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8529562Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8530417Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8531220Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8531853Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8532412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8532968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8533407Z dist init r=0, world=2 2022-11-23T03:36:56.8534430Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8535110Z warnings.warn( 2022-11-23T03:36:56.8535439Z dist init r=1, world=2 2022-11-23T03:36:56.8536533Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8537207Z warnings.warn( 2022-11-23T03:36:56.8537491Z ok (6.934s) 2022-11-23T03:36:56.8538073Z test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8538883Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29198 2022-11-23T03:36:56.8539540Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29199 2022-11-23T03:36:56.8540319Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8540885Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8541616Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8542202Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8542730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8543517Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8544065Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8544788Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8545363Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8545925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8546649Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8547362Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8547922Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8548449Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8548912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8549397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8549777Z dist init r=1, world=2 2022-11-23T03:36:56.8550629Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8551266Z warnings.warn( 2022-11-23T03:36:56.8551517Z dist init r=0, world=2 2022-11-23T03:36:56.8552362Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8552932Z warnings.warn( 2022-11-23T03:36:56.8553193Z ok (7.036s) 2022-11-23T03:36:56.8553689Z test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8554370Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29351 2022-11-23T03:36:56.8554970Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29352 2022-11-23T03:36:56.8555635Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8556073Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8556682Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8557166Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8557630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8558283Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8558752Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8559363Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8559823Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8560287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8560956Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8561674Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8562264Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8562791Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8563258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8563746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8564099Z dist init r=1, world=2 2022-11-23T03:36:56.8564373Z dist init r=0, world=2 2022-11-23T03:36:56.8565188Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:36:56.8565750Z warnings.warn( 2022-11-23T03:36:56.8566587Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8567151Z warnings.warn( 2022-11-23T03:36:56.8568039Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:36:56.8568734Z warnings.warn( 2022-11-23T03:36:56.8569718Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8570389Z warnings.warn( 2022-11-23T03:36:56.8570694Z ok (6.733s) 2022-11-23T03:36:56.8571303Z test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8572120Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29504 2022-11-23T03:36:56.8572786Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29505 2022-11-23T03:36:56.8573629Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8574205Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8574916Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8575500Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8576056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8576847Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8577408Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8578140Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8578722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8579256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8580072Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8580929Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8581648Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8582282Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8582844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8583436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8583853Z dist init r=1, world=2 2022-11-23T03:36:56.8584181Z dist init r=0, world=2 2022-11-23T03:36:56.8585161Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:36:56.8585834Z warnings.warn( 2022-11-23T03:36:56.8586770Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8587334Z warnings.warn( 2022-11-23T03:36:56.8588132Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:36:56.8588692Z warnings.warn( 2022-11-23T03:36:56.8589529Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8590143Z warnings.warn( 2022-11-23T03:36:56.8590399Z ok (6.932s) 2022-11-23T03:36:56.8590896Z test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8591581Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29657 2022-11-23T03:36:56.8592130Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29658 2022-11-23T03:36:56.8592780Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8593247Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8593882Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8594370Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8594832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8595489Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8595952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8596562Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8597049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8597523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8598180Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8598901Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8599493Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8600020Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8600487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8600976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8601356Z dist init r=1, world=2 2022-11-23T03:36:56.8601609Z dist init r=0, world=2 2022-11-23T03:36:56.8601873Z ok (4.530s) 2022-11-23T03:36:56.8602375Z test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8603060Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29800 2022-11-23T03:36:56.8603610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29801 2022-11-23T03:36:56.8604257Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8604731Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8605313Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8605794Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8606265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8606919Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8607459Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8608198Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8608743Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8609288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8610060Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8610915Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8611636Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8612467Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8613043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8613641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8614090Z dist init r=0, world=2 2022-11-23T03:36:56.8614396Z dist init r=1, world=2 2022-11-23T03:36:56.8614713Z ok (4.429s) 2022-11-23T03:36:56.8615319Z test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8616139Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29943 2022-11-23T03:36:56.8616801Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29944 2022-11-23T03:36:56.8617584Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8618150Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8618845Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8619433Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8619981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8620774Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8621334Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8622058Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8622644Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8623174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8623976Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8624831Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8625539Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8626173Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8626701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8627190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8627577Z dist init r=1, world=2 2022-11-23T03:36:56.8628360Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:36:56.8629010Z warnings.warn( 2022-11-23T03:36:56.8629852Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8630429Z warnings.warn( 2022-11-23T03:36:56.8630702Z dist init r=0, world=2 2022-11-23T03:36:56.8631512Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:36:56.8632066Z warnings.warn( 2022-11-23T03:36:56.8632954Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8633510Z warnings.warn( 2022-11-23T03:36:56.8633775Z ok (6.830s) 2022-11-23T03:36:56.8634272Z test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8634952Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30096 2022-11-23T03:36:56.8635506Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30097 2022-11-23T03:36:56.8636156Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8636625Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8637237Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8637699Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8638162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8638823Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8639284Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8639897Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8640386Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8640845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8641504Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8642211Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8642804Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8643338Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8643807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8644295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8644676Z dist init r=1, world=2 2022-11-23T03:36:56.8645460Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:36:56.8646087Z warnings.warn( 2022-11-23T03:36:56.8646928Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8647491Z warnings.warn( 2022-11-23T03:36:56.8647860Z dist init r=0, world=2 2022-11-23T03:36:56.8648733Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:36:56.8649381Z warnings.warn( 2022-11-23T03:36:56.8650367Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8651144Z warnings.warn( 2022-11-23T03:36:56.8651435Z ok (6.834s) 2022-11-23T03:36:56.8652037Z test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8652856Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30249 2022-11-23T03:36:56.8653513Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30250 2022-11-23T03:36:56.8654292Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8654864Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8655595Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8656195Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8656726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8657504Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8658062Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8658789Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8659370Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8659924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8660738Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8661576Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8662298Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8662971Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8663535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8664123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8664578Z dist init r=0, world=2 2022-11-23T03:36:56.8664913Z dist init r=1, world=2 2022-11-23T03:36:56.8665199Z ok (4.628s) 2022-11-23T03:36:56.8665796Z test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8666607Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30392 2022-11-23T03:36:56.8667244Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30393 2022-11-23T03:36:56.8667894Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8668363Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8668976Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8669437Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8669897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8670555Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8671014Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8671688Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8672180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8672646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8673332Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8674014Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8674612Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8675137Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8675605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8676098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8676477Z dist init r=1, world=2 2022-11-23T03:36:56.8676762Z dist init r=0, world=2 2022-11-23T03:36:56.8676999Z ok (4.427s) 2022-11-23T03:36:56.8677448Z test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8678072Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30535 2022-11-23T03:36:56.8678615Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30536 2022-11-23T03:36:56.8679263Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8679726Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8680340Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8680801Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8681262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8681918Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8682386Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8682993Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8683493Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8683924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8684600Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8685386Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8685980Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8686506Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8686976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8687466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8687896Z dist init r=0, world=2 2022-11-23T03:36:56.8688793Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8689471Z warnings.warn( 2022-11-23T03:36:56.8689885Z dist init r=1, world=2 2022-11-23T03:36:56.8690897Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8691589Z warnings.warn( 2022-11-23T03:36:56.8691900Z ok (6.932s) 2022-11-23T03:36:56.8692438Z test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8693152Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30688 2022-11-23T03:36:56.8693796Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30689 2022-11-23T03:36:56.8694572Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8695146Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8695873Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8696458Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8696926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8697584Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8698020Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8698627Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8699119Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8699587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8700273Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8700988Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8701584Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8702114Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8702552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8703040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8703418Z dist init r=1, world=2 2022-11-23T03:36:56.8703700Z dist init r=0, world=2 2022-11-23T03:36:56.8704544Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8705197Z warnings.warn( 2022-11-23T03:36:56.8706039Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8706585Z warnings.warn( 2022-11-23T03:36:56.8706864Z ok (6.931s) 2022-11-23T03:36:56.8707316Z test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8707948Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30841 2022-11-23T03:36:56.8708542Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30842 2022-11-23T03:36:56.8709198Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8709666Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8710245Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8710730Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8711188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8711842Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8712305Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8712917Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8713404Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8713865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8714515Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8715224Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8715818Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8716348Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8716817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8717304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8717694Z dist init r=0, world=2 2022-11-23T03:36:56.8718477Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:36:56.8719043Z warnings.warn( 2022-11-23T03:36:56.8719880Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8720457Z warnings.warn( 2022-11-23T03:36:56.8720730Z dist init r=1, world=2 2022-11-23T03:36:56.8721545Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:36:56.8722174Z warnings.warn( 2022-11-23T03:36:56.8723005Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8723579Z warnings.warn( 2022-11-23T03:36:56.8723816Z ok (6.934s) 2022-11-23T03:36:56.8724268Z test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8724889Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30994 2022-11-23T03:36:56.8725432Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30995 2022-11-23T03:36:56.8726076Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8726602Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8727227Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8727682Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8728310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8729092Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8729643Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8730375Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8730956Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8731523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8732319Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8733172Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8733879Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8734517Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8735083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8735669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8736122Z dist init r=0, world=2 2022-11-23T03:36:56.8736429Z dist init r=1, world=2 2022-11-23T03:36:56.8737263Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:36:56.8737827Z warnings.warn( 2022-11-23T03:36:56.8738663Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8739232Z warnings.warn( 2022-11-23T03:36:56.8740029Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:36:56.8740584Z warnings.warn( 2022-11-23T03:36:56.8741425Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8742087Z warnings.warn( 2022-11-23T03:36:56.8742326Z ok (6.933s) 2022-11-23T03:36:56.8742672Z test_rekey_optim_state_dict_to_names (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8743200Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31147 2022-11-23T03:36:56.8743735Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31148 2022-11-23T03:36:56.8744386Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8744855Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8745458Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8745975Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8746448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8747102Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8747573Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8748180Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8748668Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8749131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8749808Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8750500Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8751097Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8751628Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8752095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8752582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8752964Z dist init r=0, world=2 2022-11-23T03:36:56.8753804Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8754344Z warnings.warn( 2022-11-23T03:36:56.8754624Z dist init r=1, world=2 2022-11-23T03:36:56.8755481Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8756047Z warnings.warn( 2022-11-23T03:36:56.8756315Z ok (7.034s) 2022-11-23T03:36:56.8756735Z test_save_load_without_0th_param_state_state_dict_type_StateDictType_FULL_STATE_DICT (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8757345Z Tests saving and loading an optim state dict for Adam optimizer (i.e. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31300 2022-11-23T03:36:56.8757893Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31301 2022-11-23T03:36:56.8758511Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8758980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8759605Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8760174Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8760640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8761298Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8761769Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8762350Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8762833Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8763295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8764038Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8764769Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8765364Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8765894Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8766341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8766825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8767200Z dist init r=0, world=2 2022-11-23T03:36:56.8767478Z dist init r=1, world=2 2022-11-23T03:36:56.8767801Z ok (6.734s) 2022-11-23T03:36:56.8768227Z test_save_load_without_0th_param_state_state_dict_type_StateDictType_SHARDED_STATE_DICT (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8768934Z Tests saving and loading an optim state dict for Adam optimizer (i.e. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31453 2022-11-23T03:36:56.8769569Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31454 2022-11-23T03:36:56.8770352Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8770906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8771633Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8772234Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8772786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8773562Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8774102Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8774830Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8775412Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8775967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8776777Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8777635Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8778352Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8778987Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8779528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8780202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8780660Z dist init r=0, world=2 2022-11-23T03:36:56.8781639Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:36:56.8782312Z warnings.warn( 2022-11-23T03:36:56.8782637Z dist init r=1, world=2 2022-11-23T03:36:56.8783607Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:36:56.8784243Z warnings.warn( 2022-11-23T03:36:56.8784557Z ok (6.740s) 2022-11-23T03:36:56.8785069Z test_scatter_full_optim_state_dict_nested_halve_world_size (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8785955Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31606 2022-11-23T03:36:56.8786597Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31607 2022-11-23T03:36:56.8787236Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8787705Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8788288Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8788774Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8789235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8789895Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8790365Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8790968Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8791456Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8791892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8792562Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8793280Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8793879Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8794405Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8794878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8795363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8795868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T03:36:56.8796341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T03:36:56.8797019Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:36:56.8797734Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:36:56.8798725Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8799370Z warnings.warn( 2022-11-23T03:36:56.8799769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T03:36:56.8800727Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8801295Z warnings.warn( 2022-11-23T03:36:56.8801690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T03:36:56.8802337Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T03:36:56.8803049Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T03:36:56.8803523Z dist init r=1, world=2 2022-11-23T03:36:56.8803809Z dist init r=0, world=2 2022-11-23T03:36:56.8804080Z ok (7.036s) 2022-11-23T03:36:56.8804533Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8805330Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31769 2022-11-23T03:36:56.8805863Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31770 2022-11-23T03:36:56.8806504Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8806970Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8807577Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8808117Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8808624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8809400Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8809933Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8810668Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8811243Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8811790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8812600Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8813462Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8814178Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8814815Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8815337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8815930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8816378Z dist init r=1, world=2 2022-11-23T03:36:56.8817273Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8817844Z warnings.warn( 2022-11-23T03:36:56.8818124Z dist init r=0, world=2 2022-11-23T03:36:56.8819051Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8819591Z warnings.warn( 2022-11-23T03:36:56.8819858Z ok (7.134s) 2022-11-23T03:36:56.8820308Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8821093Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31922 2022-11-23T03:36:56.8821650Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31923 2022-11-23T03:36:56.8822290Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8822756Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8823425Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8823889Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8824355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8825015Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8825484Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8826093Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8826577Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8827047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8827707Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8828416Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8829002Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8829529Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8829998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8830489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8830867Z dist init r=1, world=2 2022-11-23T03:36:56.8831687Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8832258Z warnings.warn( 2022-11-23T03:36:56.8832529Z dist init r=0, world=2 2022-11-23T03:36:56.8833375Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8833951Z warnings.warn( 2022-11-23T03:36:56.8834220Z ok (7.237s) 2022-11-23T03:36:56.8834671Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8835460Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32075 2022-11-23T03:36:56.8835988Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32076 2022-11-23T03:36:56.8836712Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8837178Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8837792Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8838273Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8838739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8839393Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8839863Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8840441Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8840982Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8841456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8842141Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8842853Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8843451Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8843982Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8844426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8844918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8845291Z dist init r=0, world=2 2022-11-23T03:36:56.8846135Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8846706Z warnings.warn( 2022-11-23T03:36:56.8846980Z dist init r=1, world=2 2022-11-23T03:36:56.8847937Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8848542Z warnings.warn( 2022-11-23T03:36:56.8848822Z ok (7.139s) 2022-11-23T03:36:56.8849353Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8850314Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32228 2022-11-23T03:36:56.8850987Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32229 2022-11-23T03:36:56.8851772Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8852341Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8853076Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8853627Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8854172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8854966Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8855525Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8856357Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8856897Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8857361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8858014Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8858731Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8859325Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8859847Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8860313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8860864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8861246Z dist init r=0, world=2 2022-11-23T03:36:56.8862096Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8862640Z warnings.warn( 2022-11-23T03:36:56.8862915Z dist init r=1, world=2 2022-11-23T03:36:56.8863755Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8864325Z warnings.warn( 2022-11-23T03:36:56.8864589Z ok (7.336s) 2022-11-23T03:36:56.8865040Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8865829Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32381 2022-11-23T03:36:56.8866352Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32382 2022-11-23T03:36:56.8866993Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8867458Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8868064Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8868552Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8869013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8869668Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8870143Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8870725Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8871215Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8871674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8872349Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8873072Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8873680Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8874218Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8874741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8875236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8875612Z dist init r=0, world=2 2022-11-23T03:36:56.8875889Z dist init r=1, world=2 2022-11-23T03:36:56.8876737Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8877311Z warnings.warn( 2022-11-23T03:36:56.8878148Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8878769Z warnings.warn( 2022-11-23T03:36:56.8879010Z ok (7.234s) 2022-11-23T03:36:56.8879461Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8880245Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32534 2022-11-23T03:36:56.8880799Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32535 2022-11-23T03:36:56.8881444Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8881914Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8882522Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8882984Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8883448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8884103Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8884565Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8885181Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8885657Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8886121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8886798Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8887490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8888149Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8888735Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8889285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8889852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8890310Z dist init r=0, world=2 2022-11-23T03:36:56.8891326Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8892005Z warnings.warn( 2022-11-23T03:36:56.8892300Z dist init r=1, world=2 2022-11-23T03:36:56.8893313Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8894089Z warnings.warn( 2022-11-23T03:36:56.8894399Z ok (7.233s) 2022-11-23T03:36:56.8894940Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8895885Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32687 2022-11-23T03:36:56.8896557Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32688 2022-11-23T03:36:56.8897217Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8897683Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8898351Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8898859Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8899326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8899984Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8900448Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8901031Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8901523Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8901982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8902666Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8903385Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8903977Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8904505Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8904975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8905434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8905802Z dist init r=1, world=2 2022-11-23T03:36:56.8906644Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8907225Z warnings.warn( 2022-11-23T03:36:56.8907502Z dist init r=0, world=2 2022-11-23T03:36:56.8908345Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8908912Z warnings.warn( 2022-11-23T03:36:56.8909152Z ok (7.238s) 2022-11-23T03:36:56.8909596Z test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8910381Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32840 2022-11-23T03:36:56.8910939Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32841 2022-11-23T03:36:56.8911583Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8912147Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8912760Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8913253Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8913690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8914347Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8914816Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8915426Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8915912Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8916439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8917131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8917822Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8918419Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8918952Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8919423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8919905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8920280Z dist init r=0, world=2 2022-11-23T03:36:56.8920560Z dist init r=1, world=2 2022-11-23T03:36:56.8921384Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8921954Z warnings.warn( 2022-11-23T03:36:56.8922791Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8923352Z warnings.warn( 2022-11-23T03:36:56.8923619Z ok (6.935s) 2022-11-23T03:36:56.8923977Z test_scatter_full_optim_state_dict_transformer (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8924677Z Tests :meth:`scatter_full_optim_state_dict` for an FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32993 2022-11-23T03:36:56.8925228Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32994 2022-11-23T03:36:56.8925847Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8926311Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8926918Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8927404Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8927986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8928649Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8929118Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8929698Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8930313Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8930781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8931469Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8932184Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8932778Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8933307Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8933775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8934233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8934797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T03:36:56.8935309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T03:36:56.8935987Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:36:56.8936702Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:36:56.8937240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T03:36:56.8937748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T03:36:56.8938401Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T03:36:56.8939116Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T03:36:56.8939556Z dist init r=1, world=2 2022-11-23T03:36:56.8939830Z dist init r=0, world=2 2022-11-23T03:36:56.8940099Z ok (7.433s) 2022-11-23T03:36:56.8940471Z test_shard_full_optim_state_dict_nested_halve_world_size (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8941204Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33156 2022-11-23T03:36:56.8941742Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33157 2022-11-23T03:36:56.8942384Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8942855Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8943460Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8943949Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8944423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8945086Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8945553Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8946135Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8946619Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8947096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8947777Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8948492Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8949192Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8949724Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8950170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8950657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8951150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T03:36:56.8951650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T03:36:56.8952325Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:36:56.8953089Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:36:56.8954084Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8954202Z warnings.warn( 2022-11-23T03:36:56.8954447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T03:36:56.8955127Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8955244Z warnings.warn( 2022-11-23T03:36:56.8955483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T03:36:56.8955869Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T03:36:56.8956284Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T03:36:56.8956403Z dist init r=1, world=2 2022-11-23T03:36:56.8956520Z dist init r=0, world=2 2022-11-23T03:36:56.8956626Z ok (7.234s) 2022-11-23T03:36:56.8956936Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8957409Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33319 2022-11-23T03:36:56.8957634Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33320 2022-11-23T03:36:56.8958026Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8958208Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8958607Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8958800Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8959040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8959430Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8959609Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8960007Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8960201Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8960508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8960929Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8961338Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8961635Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8961930Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8962133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8962359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8962478Z dist init r=1, world=2 2022-11-23T03:36:56.8963208Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8963330Z warnings.warn( 2022-11-23T03:36:56.8963448Z dist init r=0, world=2 2022-11-23T03:36:56.8964133Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8964249Z warnings.warn( 2022-11-23T03:36:56.8964357Z ok (7.335s) 2022-11-23T03:36:56.8964661Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8965136Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33472 2022-11-23T03:36:56.8965367Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33473 2022-11-23T03:36:56.8965754Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8965934Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8966330Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8966525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8966766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8967150Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8967332Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8967790Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8967994Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8968233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8968619Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8969031Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8969321Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8969611Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8969844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8970071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8970295Z dist init r=0, world=2 2022-11-23T03:36:56.8970977Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8971092Z warnings.warn( 2022-11-23T03:36:56.8971209Z dist init r=1, world=2 2022-11-23T03:36:56.8971886Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8972002Z warnings.warn( 2022-11-23T03:36:56.8972109Z ok (7.134s) 2022-11-23T03:36:56.8972406Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8972931Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33625 2022-11-23T03:36:56.8973164Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33626 2022-11-23T03:36:56.8973560Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8973743Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8974144Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8974338Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8974577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8974965Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8975161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8975531Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8975726Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8975966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8976374Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8976799Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8977088Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8977379Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8977620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8977851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8977969Z dist init r=1, world=2 2022-11-23T03:36:56.8978085Z dist init r=0, world=2 2022-11-23T03:36:56.8978763Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8978882Z warnings.warn( 2022-11-23T03:36:56.8979566Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8979681Z warnings.warn( 2022-11-23T03:36:56.8979847Z ok (7.035s) 2022-11-23T03:36:56.8980151Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8980626Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33778 2022-11-23T03:36:56.8980851Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33779 2022-11-23T03:36:56.8981241Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8981421Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8981822Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8981987Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8982276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8982677Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8982856Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8983249Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8983444Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8983681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8984087Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8984503Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8984803Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8985093Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8985320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8985552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8985671Z dist init r=1, world=2 2022-11-23T03:36:56.8986350Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8986467Z warnings.warn( 2022-11-23T03:36:56.8986588Z dist init r=0, world=2 2022-11-23T03:36:56.8987268Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8987388Z warnings.warn( 2022-11-23T03:36:56.8987494Z ok (7.436s) 2022-11-23T03:36:56.8987786Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8988250Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33931 2022-11-23T03:36:56.8988466Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33932 2022-11-23T03:36:56.8988827Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8989001Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8989401Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8989651Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8989884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8990270Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8990445Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8990835Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8991022Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8991254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8991659Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8992112Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8992407Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8992693Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.8992919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.8993141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.8993252Z dist init r=1, world=2 2022-11-23T03:36:56.8993919Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8994031Z warnings.warn( 2022-11-23T03:36:56.8994146Z dist init r=0, world=2 2022-11-23T03:36:56.8994823Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.8994930Z warnings.warn( 2022-11-23T03:36:56.8995012Z ok (7.135s) 2022-11-23T03:36:56.8995303Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.8995762Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34084 2022-11-23T03:36:56.8995977Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34085 2022-11-23T03:36:56.8996359Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8996540Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8996930Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8997122Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8997356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.8997737Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.8997915Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.8998309Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.8998498Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.8998733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.8999205Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8999611Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.8999898Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.9000182Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.9000407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.9000628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.9000744Z dist init r=0, world=2 2022-11-23T03:36:56.9001482Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.9001600Z warnings.warn( 2022-11-23T03:36:56.9001690Z dist init r=1, world=2 2022-11-23T03:36:56.9002363Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.9002472Z warnings.warn( 2022-11-23T03:36:56.9002568Z ok (7.134s) 2022-11-23T03:36:56.9002860Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.9003318Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34237 2022-11-23T03:36:56.9003537Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34238 2022-11-23T03:36:56.9003920Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.9004092Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.9004486Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.9004674Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.9004912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.9005290Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.9005464Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.9005851Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.9006045Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.9006275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.9006681Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.9007089Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.9007373Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.9007659Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.9007948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.9008150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.9008326Z dist init r=0, world=2 2022-11-23T03:36:56.9009006Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.9009118Z warnings.warn( 2022-11-23T03:36:56.9009228Z dist init r=1, world=2 2022-11-23T03:36:56.9009898Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.9010005Z warnings.warn( 2022-11-23T03:36:56.9010104Z ok (7.937s) 2022-11-23T03:36:56.9010395Z test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.9010910Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34390 2022-11-23T03:36:56.9011139Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34391 2022-11-23T03:36:56.9011522Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.9011695Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.9012084Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.9012272Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.9012506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.9012887Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.9013065Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.9013452Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.9013639Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.9013868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.9014296Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.9014707Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.9014972Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.9015256Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.9015486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.9015715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.9015830Z dist init r=1, world=2 2022-11-23T03:36:56.9016712Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.9016805Z warnings.warn( 2022-11-23T03:36:56.9016920Z dist init r=0, world=2 2022-11-23T03:36:56.9017588Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.9017699Z warnings.warn( 2022-11-23T03:36:56.9017801Z ok (6.933s) 2022-11-23T03:36:56.9018011Z test_shard_full_optim_state_dict_transformer (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.9018525Z Tests :meth:`shard_full_optim_state_dict` for an FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34543 2022-11-23T03:36:56.9018754Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34544 2022-11-23T03:36:56.9019181Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.9019355Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.9019746Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.9019936Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.9020169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.9020600Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.9020797Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.9021194Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.9021409Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.9021645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.9022054Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.9022462Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.9022746Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.9023047Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.9023250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.9023474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.9023710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T03:36:56.9023962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T03:36:56.9024371Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:36:56.9024771Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:36:56.9025008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T03:36:56.9025244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T03:36:56.9025665Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T03:36:56.9026066Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T03:36:56.9026183Z dist init r=1, world=2 2022-11-23T03:36:56.9026314Z dist init r=0, world=2 2022-11-23T03:36:56.9026416Z ok (7.334s) 2022-11-23T03:36:56.9026728Z test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_FULL_STATE_DICT_add_to_fsdp_module_False (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.9027029Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34706 2022-11-23T03:36:56.9027255Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34707 2022-11-23T03:36:56.9027714Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.9027891Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.9028280Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.9028491Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.9028727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.9029109Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.9029265Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.9029664Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.9029904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.9030147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.9030556Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.9030958Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.9031269Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.9060505Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.9060785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.9061015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.9061127Z dist init r=0, world=2 2022-11-23T03:36:56.9061237Z dist init r=1, world=2 2022-11-23T03:36:56.9062015Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.9062125Z warnings.warn( 2022-11-23T03:36:56.9062806Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.9062913Z warnings.warn( 2022-11-23T03:36:56.9062992Z ok (7.034s) 2022-11-23T03:36:56.9063303Z test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_FULL_STATE_DICT_add_to_fsdp_module_True (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.9063607Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34859 2022-11-23T03:36:56.9063828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34860 2022-11-23T03:36:56.9064214Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.9064387Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.9064782Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.9064967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.9065200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.9065583Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.9065755Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.9066294Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.9066481Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.9066712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.9067118Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.9067519Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.9067803Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.9068086Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.9068308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.9068592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.9068709Z dist init r=0, world=2 2022-11-23T03:36:56.9068801Z dist init r=1, world=2 2022-11-23T03:36:56.9069485Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.9069594Z warnings.warn( 2022-11-23T03:36:56.9070263Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.9070369Z warnings.warn( 2022-11-23T03:36:56.9070461Z ok (6.634s) 2022-11-23T03:36:56.9070779Z test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_SHARDED_STATE_DICT_add_to_fsdp_module_False (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.9071087Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35012 2022-11-23T03:36:56.9071302Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35013 2022-11-23T03:36:56.9071687Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.9071858Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.9072251Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.9072437Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.9072667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.9073051Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.9073221Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.9073611Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.9073800Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.9074030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.9074433Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.9074838Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.9075120Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.9075473Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.9075675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.9075895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.9076006Z dist init r=0, world=2 2022-11-23T03:36:56.9076653Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:36:56.9076761Z warnings.warn( 2022-11-23T03:36:56.9077428Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.9077534Z warnings.warn( 2022-11-23T03:36:56.9077694Z dist init r=1, world=2 2022-11-23T03:36:56.9078336Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:36:56.9078442Z warnings.warn( 2022-11-23T03:36:56.9079108Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.9079215Z warnings.warn( 2022-11-23T03:36:56.9079312Z ok (6.734s) 2022-11-23T03:36:56.9079623Z test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_SHARDED_STATE_DICT_add_to_fsdp_module_True (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.9079929Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35165 2022-11-23T03:36:56.9080145Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35166 2022-11-23T03:36:56.9080526Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.9080698Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.9081089Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.9081272Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.9081502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.9081880Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.9082053Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.9082449Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.9082615Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.9082844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.9083244Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.9083642Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.9083926Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.9084206Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.9084427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.9084711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.9084819Z dist init r=0, world=2 2022-11-23T03:36:56.9084930Z dist init r=1, world=2 2022-11-23T03:36:56.9085582Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:36:56.9085689Z warnings.warn( 2022-11-23T03:36:56.9086365Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.9086472Z warnings.warn( 2022-11-23T03:36:56.9087155Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:36:56.9087270Z warnings.warn( 2022-11-23T03:36:56.9088015Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1232: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality. 2022-11-23T03:36:56.9088125Z warnings.warn( 2022-11-23T03:36:56.9088223Z ok (6.735s) 2022-11-23T03:36:56.9088399Z test_use_orig_params_error (__main__.TestFSDPOptimState) 2022-11-23T03:36:56.9088774Z Tests that the optimizer state checkpointing APIs raise an error ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35318 2022-11-23T03:36:56.9089029Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35319 2022-11-23T03:36:56.9089489Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.9089675Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.9090149Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.9090369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.9090645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:56.9091100Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:56.9091303Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:56.9091775Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:56.9092000Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:56.9092278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:56.9092768Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.9093253Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:56.9093593Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.9093932Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:56.9094200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:56.9094466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:56.9094602Z dist init r=1, world=2 2022-11-23T03:36:56.9094729Z dist init r=0, world=2 2022-11-23T03:36:56.9094859Z ok (6.835s) 2022-11-23T03:36:56.9094869Z 2022-11-23T03:36:56.9095310Z ---------------------------------------------------------------------- 2022-11-23T03:36:56.9095451Z Ran 53 tests in 362.794s 2022-11-23T03:36:56.9095458Z 2022-11-23T03:36:56.9095544Z OK 2022-11-23T03:36:56.9095575Z 2022-11-23T03:36:56.9095698Z Generating XML reports... 2022-11-23T03:36:56.9096256Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state/TEST-TestFSDPOptimState-20221123033051.xml 2022-11-23T03:36:56.9096264Z 2022-11-23T03:36:56.9096793Z ##[endgroup] 2022-11-23T03:36:56.9097384Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_optim_state (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_optim_state_yxx606yx) 2022-11-23T03:36:56.9097391Z 2022-11-23T03:36:56.9097690Z Running distributed/fsdp/test_fsdp_multiple_forward ... [2022-11-23 03:36:56.832033] 2022-11-23T03:36:56.9098269Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_forward.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:36:56.832705] 2022-11-23T03:37:07.9630934Z 2022-11-23T03:37:07.9632814Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_multiple_forward 2022-11-23T03:37:07.9639334Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_forward (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_multiple_forward_po9bqcec) 2022-11-23T03:37:07.9641620Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward 2022-11-23T03:37:07.9642394Z 2022-11-23T03:37:07.9642637Z Running tests... 2022-11-23T03:37:07.9643806Z ---------------------------------------------------------------------- 2022-11-23T03:37:07.9645156Z test_multi_forward (__main__.TestMultiForward) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35538 2022-11-23T03:37:07.9646527Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35539 2022-11-23T03:37:07.9648458Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:07.9649708Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:07.9651320Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:07.9652580Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:07.9653742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:37:07.9655534Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:07.9656732Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:07.9658320Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:07.9659583Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:07.9660802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:37:07.9662599Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:37:07.9664486Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:37:07.9665994Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:37:07.9667360Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:37:07.9668575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:37:07.9669836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:37:07.9671111Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:37:07.9672986Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:37:07.9673969Z dist init r=0, world=2 2022-11-23T03:37:07.9677357Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:37:07.9679426Z warnings.warn( 2022-11-23T03:37:07.9680073Z dist init r=1, world=2 2022-11-23T03:37:07.9683564Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:37:07.9685586Z warnings.warn( 2022-11-23T03:37:07.9686185Z ok (7.133s) 2022-11-23T03:37:07.9686545Z 2022-11-23T03:37:07.9687275Z ---------------------------------------------------------------------- 2022-11-23T03:37:07.9688237Z Ran 1 test in 7.134s 2022-11-23T03:37:07.9688658Z 2022-11-23T03:37:07.9688894Z OK 2022-11-23T03:37:07.9689234Z 2022-11-23T03:37:07.9689539Z Generating XML reports... 2022-11-23T03:37:07.9691188Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward/TEST-TestMultiForward-20221123033658.xml 2022-11-23T03:37:07.9692096Z 2022-11-23T03:37:07.9692886Z ##[endgroup] 2022-11-23T03:37:07.9694678Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_forward (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_multiple_forward_po9bqcec) 2022-11-23T03:37:07.9695677Z 2022-11-23T03:37:07.9696400Z Running distributed/fsdp/test_fsdp_misc ... [2022-11-23 03:37:07.963370] 2022-11-23T03:37:07.9698251Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_misc.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:37:07.964032] 2022-11-23T03:38:37.5111639Z 2022-11-23T03:38:37.5112611Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_misc 2022-11-23T03:38:37.5114736Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_misc (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_misc_uh73m7ud) 2022-11-23T03:38:37.5120163Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_misc 2022-11-23T03:38:37.5121036Z 2022-11-23T03:38:37.5121284Z Running tests... 2022-11-23T03:38:37.5122510Z ---------------------------------------------------------------------- 2022-11-23T03:38:37.5123672Z test_cpu_init_with_sync_module_states (__main__.TestFSDPMisc) 2022-11-23T03:38:37.5125008Z Tests that passing ``sync_module_states=True`` raises an error for ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35758 2022-11-23T03:38:37.5127210Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35759 2022-11-23T03:38:37.5129902Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:37.5131146Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:37.5132885Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:37.5134586Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:37.5136111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:38:37.5138611Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:37.5140877Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:37.5143346Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:37.5145079Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:37.5146642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:38:37.5149182Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:38:37.5151598Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:38:37.5153153Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:38:37.5154780Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:38:37.5155997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:38:37.5157202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:38:37.5158096Z dist init r=0, world=2 2022-11-23T03:38:37.5161371Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:38:37.5163442Z warnings.warn( 2022-11-23T03:38:37.5164083Z dist init r=1, world=2 2022-11-23T03:38:37.5167292Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:38:37.5169520Z warnings.warn( 2022-11-23T03:38:37.5170170Z ok (5.018s) 2022-11-23T03:38:37.5170910Z test_device_id_auto_wrap (__main__.TestFSDPMisc) 2022-11-23T03:38:37.5172188Z Tests that ``auto_wrap_policy`` propagates ``device_id`` to all ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35901 2022-11-23T03:38:37.5173522Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35902 2022-11-23T03:38:37.5175198Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:37.5176394Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:37.5177915Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:37.5179149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:37.5180298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:38:37.5181967Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:37.5183109Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:37.5184653Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:37.5185872Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:37.5187025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:38:37.5189184Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:38:37.5191029Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:38:37.5192524Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:38:37.5193863Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:38:37.5194988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:38:37.5196168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:38:37.5197097Z dist init r=1, world=2 2022-11-23T03:38:37.5197743Z dist init r=0, world=2 2022-11-23T03:38:37.5198381Z ok (4.629s) 2022-11-23T03:38:37.5199151Z test_fsdp_cpu_init_stays_on_cpu (__main__.TestFSDPMisc) 2022-11-23T03:38:37.5200609Z Tests that passing a CPU module to FSDP preserves that the wrapped ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36044 2022-11-23T03:38:37.5202027Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36045 2022-11-23T03:38:37.5203676Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:37.5204852Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:37.5206384Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:37.5207589Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:37.5209169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:38:37.5210827Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:37.5212030Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:37.5213587Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:37.5214802Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:37.5215962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:38:37.5217705Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:38:37.5219546Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:38:37.5221051Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:38:37.5222350Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:38:37.5222935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:38:37.5223436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:38:37.5223794Z dist init r=0, world=2 2022-11-23T03:38:37.5224063Z dist init r=1, world=2 2022-11-23T03:38:37.5224315Z ok (6.432s) 2022-11-23T03:38:37.5224602Z test_fsdp_device_id_cpu_offload (__main__.TestFSDPMisc) 2022-11-23T03:38:37.5225094Z Ensures that even if device_id is specified but we have ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36197 2022-11-23T03:38:37.5225609Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36198 2022-11-23T03:38:37.5226246Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:37.5226702Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:37.5227295Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:37.5227849Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:37.5228278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:38:37.5228924Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:37.5229360Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:37.5229959Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:37.5230432Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:37.5230872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:38:37.5231586Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:38:37.5232288Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:38:37.5232833Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:38:37.5233333Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:38:37.5233779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:38:37.5234255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:38:37.5234617Z dist init r=1, world=2 2022-11-23T03:38:37.5234871Z dist init r=0, world=2 2022-11-23T03:38:37.5235105Z ok (4.429s) 2022-11-23T03:38:37.5235417Z test_fsdp_device_id_use_index_False (__main__.TestFSDPMisc) 2022-11-23T03:38:37.5235870Z Tests the FSDP ``device_id`` argument: ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36340 2022-11-23T03:38:37.5236378Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36341 2022-11-23T03:38:37.5237009Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:37.5237463Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:37.5238040Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:37.5238484Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:37.5238938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:38:37.5239583Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:37.5240033Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:37.5240630Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:37.5241085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:37.5241533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:38:37.5242190Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:38:37.5242864Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:38:37.5243434Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:38:37.5243931Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:38:37.5244396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:38:37.5244883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:38:37.5245326Z dist init r=1, world=2 2022-11-23T03:38:37.5245608Z dist init r=0, world=2 2022-11-23T03:38:37.5245826Z ok (4.431s) 2022-11-23T03:38:37.5246119Z test_fsdp_device_id_use_index_True (__main__.TestFSDPMisc) 2022-11-23T03:38:37.5246585Z Tests the FSDP ``device_id`` argument: ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36483 2022-11-23T03:38:37.5247071Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36484 2022-11-23T03:38:37.5247759Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:37.5248243Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:37.5248828Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:37.5249365Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:37.5249836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:38:37.5250579Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:37.5251125Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:37.5251839Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:37.5252412Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:37.5252937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:38:37.5253759Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:38:37.5254611Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:38:37.5255319Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:38:37.5255955Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:38:37.5256505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:38:37.5257080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:38:37.5257520Z dist init r=1, world=2 2022-11-23T03:38:37.5257814Z dist init r=0, world=2 2022-11-23T03:38:37.5258123Z ok (4.427s) 2022-11-23T03:38:37.5258724Z test_fsdp_module_no_compute_grad_use_second_layer_False_sharding_strategy_None (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36626 2022-11-23T03:38:37.5259439Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36627 2022-11-23T03:38:37.5260207Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:37.5260752Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:37.5261481Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:37.5262040Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:37.5262598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:38:37.5263276Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:37.5263740Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:37.5264346Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:37.5264831Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:37.5265365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:38:37.5266020Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:38:37.5266946Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:38:37.5267534Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:38:37.5268064Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:38:37.5268508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:38:37.5268989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:38:37.5269358Z dist init r=0, world=2 2022-11-23T03:38:37.5269629Z dist init r=1, world=2 2022-11-23T03:38:37.5269904Z ok (6.532s) 2022-11-23T03:38:37.5270502Z test_fsdp_module_no_compute_grad_use_second_layer_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36779 2022-11-23T03:38:37.5271148Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36780 2022-11-23T03:38:37.5271769Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:37.5272234Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:37.5272838Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:37.5273319Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:37.5273787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:38:37.5274450Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:37.5274913Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:37.5275491Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:37.5275980Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:37.5276413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:38:37.5277079Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:38:37.5277771Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:38:37.5278361Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:38:37.5278892Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:38:37.5279343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:38:37.5279798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:38:37.5280158Z dist init r=0, world=2 2022-11-23T03:38:37.5280429Z dist init r=1, world=2 2022-11-23T03:38:37.5280690Z ok (6.531s) 2022-11-23T03:38:37.5281197Z test_fsdp_module_no_compute_grad_use_second_layer_True_sharding_strategy_None (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36932 2022-11-23T03:38:37.5281780Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36933 2022-11-23T03:38:37.5282396Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:37.5282861Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:37.5283537Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:37.5284020Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:37.5284486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:38:37.5285143Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:37.5285610Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:37.5286224Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:37.5286689Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:37.5287154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:38:37.5287961Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:38:37.5288699Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:38:37.5289286Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:38:37.5289819Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:38:37.5290323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:38:37.5290870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:38:37.5291307Z dist init r=1, world=2 2022-11-23T03:38:37.5291621Z dist init r=0, world=2 2022-11-23T03:38:37.5291930Z ok (7.233s) 2022-11-23T03:38:37.5292568Z test_fsdp_module_no_compute_grad_use_second_layer_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37085 2022-11-23T03:38:37.5293322Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37086 2022-11-23T03:38:37.5294088Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:37.5294615Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:37.5295335Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:37.5295916Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:37.5296461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:38:37.5297235Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:37.5297799Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:37.5298528Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:37.5299085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:37.5299632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:38:37.5300437Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:38:37.5301284Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:38:37.5301986Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:38:37.5302621Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:38:37.5303112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:38:37.5303675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:38:37.5304021Z dist init r=0, world=2 2022-11-23T03:38:37.5304302Z dist init r=1, world=2 2022-11-23T03:38:37.5304566Z ok (6.437s) 2022-11-23T03:38:37.5305002Z test_fsdp_namedtuple (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37238 2022-11-23T03:38:37.5305537Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37239 2022-11-23T03:38:37.5306184Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:37.5306619Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:37.5307227Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:37.5307713Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:37.5308232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:38:37.5308897Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:37.5309368Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:37.5309982Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:37.5310438Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:37.5310906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:38:37.5311574Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:38:37.5312276Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:38:37.5312874Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:38:37.5313397Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:38:37.5313855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:38:37.5314344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:38:37.5314684Z dist init r=1, world=2 2022-11-23T03:38:37.5314954Z dist init r=0, world=2 2022-11-23T03:38:37.5315215Z ok (4.529s) 2022-11-23T03:38:37.5315660Z test_fsdp_not_all_outputs_used_in_loss (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37381 2022-11-23T03:38:37.5316204Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37382 2022-11-23T03:38:37.5316852Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:37.5317302Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:37.5317906Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:37.5318391Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:37.5318851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:38:37.5319498Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:37.5319959Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:37.5320564Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:37.5321021Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:37.5321484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:38:37.5322252Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:38:37.5322958Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:38:37.5323552Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:38:37.5324073Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:38:37.5324532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:38:37.5325013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:38:37.5325357Z dist init r=1, world=2 2022-11-23T03:38:37.5326247Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:38:37.5326874Z warnings.warn(message, UserWarning) 2022-11-23T03:38:37.5327182Z dist init r=0, world=2 2022-11-23T03:38:37.5328198Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:38:37.5328808Z warnings.warn(message, UserWarning) 2022-11-23T03:38:37.5329103Z ok (6.836s) 2022-11-23T03:38:37.5329405Z test_fsdp_same_model_across_ranks (__main__.TestFSDPMisc) 2022-11-23T03:38:37.5329925Z FSDP broadcasts model from rank 0 to ensure it starts off with the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37534 2022-11-23T03:38:37.5330538Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37535 2022-11-23T03:38:37.5331314Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:37.5331871Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:37.5332593Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:37.5333177Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:37.5333726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:38:37.5334469Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:37.5335031Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:37.5335753Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:37.5336336Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:37.5336885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:38:37.5337678Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:38:37.5338541Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:38:37.5339216Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:38:37.5339837Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:38:37.5340378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:38:37.5340953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:38:37.5341399Z dist init r=0, world=2 2022-11-23T03:38:37.5341828Z dist init r=1, world=2 2022-11-23T03:38:37.5342110Z ok (4.530s) 2022-11-23T03:38:37.5342513Z test_module_device_mismatches_device_id (__main__.TestFSDPMisc) 2022-11-23T03:38:37.5343071Z Tests that specifying a ``device_id`` argument to FSDP for a GPU ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37677 2022-11-23T03:38:37.5343626Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37678 2022-11-23T03:38:37.5344272Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:37.5344749Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:37.5345355Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:37.5345839Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:37.5346327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:38:37.5346994Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:37.5347462Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:37.5348065Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:37.5348546Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:37.5348999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:38:37.5349678Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:38:37.5350365Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:38:37.5350961Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:38:37.5351490Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:38:37.5351938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:38:37.5352420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:38:37.5352785Z dist init r=1, world=2 2022-11-23T03:38:37.5353061Z dist init r=0, world=2 2022-11-23T03:38:37.5353295Z ok (4.429s) 2022-11-23T03:38:37.5353619Z test_multi_device_not_supported (__main__.TestFSDPMisc) 2022-11-23T03:38:37.5354313Z Tests that wrapping a multi-device module (i.e. with submodules on ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37820 2022-11-23T03:38:37.5354871Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37821 2022-11-23T03:38:37.5355514Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:37.5355981Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:37.5356593Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:37.5357045Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:37.5357500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:38:37.5358139Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:37.5358599Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:37.5359205Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:37.5359690Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:37.5360212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:38:37.5360861Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:38:37.5361546Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:38:37.5362118Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:38:37.5362640Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:38:37.5363101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:38:37.5363585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:38:37.5363952Z dist init r=1, world=2 2022-11-23T03:38:37.5364201Z dist init r=0, world=2 2022-11-23T03:38:37.5364465Z ok (4.429s) 2022-11-23T03:38:37.5364826Z test_no_params (__main__.TestFSDPMisc) 2022-11-23T03:38:37.5365321Z Test that device_id and cpu init work if module has no params ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37963 2022-11-23T03:38:37.5365865Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37964 2022-11-23T03:38:37.5366499Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:37.5366956Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:37.5367526Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:37.5368133Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:37.5368642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:38:37.5369430Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:37.5369990Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:37.5370701Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:37.5371249Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:37.5371745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:38:37.5372530Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:38:37.5373374Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:38:37.5374073Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:38:37.5374711Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:38:37.5375274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:38:37.5375830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:38:37.5376233Z dist init r=1, world=2 2022-11-23T03:38:37.5376563Z dist init r=0, world=2 2022-11-23T03:38:37.5376869Z ok (4.729s) 2022-11-23T03:38:37.5377050Z 2022-11-23T03:38:37.5377400Z ---------------------------------------------------------------------- 2022-11-23T03:38:37.5377798Z Ran 16 tests in 85.585s 2022-11-23T03:38:37.5377985Z 2022-11-23T03:38:37.5378103Z OK 2022-11-23T03:38:37.5378267Z 2022-11-23T03:38:37.5378391Z Generating XML reports... 2022-11-23T03:38:37.5379097Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_misc/TEST-TestFSDPMisc-20221123033709.xml 2022-11-23T03:38:37.5379486Z 2022-11-23T03:38:37.5379928Z ##[endgroup] 2022-11-23T03:38:37.5380673Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_misc (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_misc_uh73m7ud) 2022-11-23T03:38:37.5381183Z 2022-11-23T03:38:37.5381523Z Running distributed/fsdp/test_fsdp_memory ... [2022-11-23 03:38:37.512670] 2022-11-23T03:38:37.5382391Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_memory.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:38:37.513389] 2022-11-23T03:39:01.5848309Z 2022-11-23T03:39:01.5849755Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_memory 2022-11-23T03:39:01.5852375Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_memory (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_memory_rlo2laqz) 2022-11-23T03:39:01.5858387Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_memory 2022-11-23T03:39:01.5859389Z 2022-11-23T03:39:01.5859702Z Running tests... 2022-11-23T03:39:01.5861533Z ---------------------------------------------------------------------- 2022-11-23T03:39:01.5863019Z test_fsdp_memory_ckpt_ckpt (__main__.TestFSDPMemory) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38173 2022-11-23T03:39:01.5864423Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38174 2022-11-23T03:39:01.5917665Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:01.5918894Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:01.5920542Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:01.5921835Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:01.5922977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:39:01.5924669Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:01.5925841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:01.5927387Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:01.5928978Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:01.5930135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:39:01.5931924Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:39:01.5933780Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:39:01.5935305Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:39:01.5936620Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:39:01.5937820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:39:01.5939038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:39:01.5939949Z dist init r=0, world=2 2022-11-23T03:39:01.5942659Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:39:01.5944139Z warnings.warn( 2022-11-23T03:39:01.5944608Z dist init r=1, world=2 2022-11-23T03:39:01.5947004Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:39:01.5948748Z warnings.warn( 2022-11-23T03:39:01.5949180Z ok (12.841s) 2022-11-23T03:39:01.5949975Z test_fsdp_memory_ckpt_no_ckpt (__main__.TestFSDPMemory) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38628 2022-11-23T03:39:01.5950947Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38629 2022-11-23T03:39:01.5952147Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:01.5952979Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:01.5954220Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:01.5955105Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:01.5955915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:39:01.5957144Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:01.5957971Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:01.5959094Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:01.5959958Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:01.5960780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:39:01.5962094Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:39:01.5963413Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:39:01.5964500Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:39:01.5965444Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:39:01.5966289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:39:01.5967172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:39:01.5967904Z dist init r=0, world=2 2022-11-23T03:39:01.5970328Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:39:01.5971819Z warnings.warn( 2022-11-23T03:39:01.5972276Z dist init r=1, world=2 2022-11-23T03:39:01.5974667Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:39:01.5976134Z warnings.warn( 2022-11-23T03:39:01.5976566Z ok (7.136s) 2022-11-23T03:39:01.5976835Z 2022-11-23T03:39:01.5977363Z ---------------------------------------------------------------------- 2022-11-23T03:39:01.5977976Z Ran 2 tests in 19.978s 2022-11-23T03:39:01.5978407Z 2022-11-23T03:39:01.5978566Z OK 2022-11-23T03:39:01.5978799Z 2022-11-23T03:39:01.5979013Z Generating XML reports... 2022-11-23T03:39:01.5980155Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_memory/TEST-TestFSDPMemory-20221123033839.xml 2022-11-23T03:39:01.5980761Z 2022-11-23T03:39:01.5981519Z ##[endgroup] 2022-11-23T03:39:01.5982693Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_memory (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_memory_rlo2laqz) 2022-11-23T03:39:01.5983333Z 2022-11-23T03:39:01.5983885Z Running distributed/fsdp/test_fsdp_ignored_modules ... [2022-11-23 03:39:01.585163] 2022-11-23T03:39:01.5985282Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_ignored_modules.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:39:01.585842] 2022-11-23T03:39:38.1709569Z 2022-11-23T03:39:38.1711654Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_ignored_modules 2022-11-23T03:39:38.1715006Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_ignored_modules (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_ignored_modules_2jqpb84t) 2022-11-23T03:39:38.1718198Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules 2022-11-23T03:39:38.1719910Z 2022-11-23T03:39:38.1720294Z Running tests... 2022-11-23T03:39:38.1721899Z ---------------------------------------------------------------------- 2022-11-23T03:39:38.1727102Z test_diff_ignored_modules_across_ranks_pass_ignored_modules_to_root_False (__main__.TestFSDPIgnoredModules) 2022-11-23T03:39:38.1728387Z Tests ignoring different modules across ranks. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38910 2022-11-23T03:39:38.1729214Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38911 2022-11-23T03:39:38.1730060Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:38.1730598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:38.1731297Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:38.1731857Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:38.1732569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:39:38.1733794Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:38.1734570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:38.1735504Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:38.1736232Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:38.1737194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:39:38.1738650Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:39:38.1740118Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:39:38.1741372Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:39:38.1742454Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:39:38.1743502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:39:38.1744609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:39:38.1745460Z dist init r=0, world=2 2022-11-23T03:39:38.1746121Z dist init r=1, world=2 2022-11-23T03:39:38.1747042Z ok (7.222s) 2022-11-23T03:39:38.1748026Z test_diff_ignored_modules_across_ranks_pass_ignored_modules_to_root_True (__main__.TestFSDPIgnoredModules) 2022-11-23T03:39:38.1749089Z Tests ignoring different modules across ranks. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39063 2022-11-23T03:39:38.1750048Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39064 2022-11-23T03:39:38.1751254Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:38.1752100Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:38.1753217Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:38.1754098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:38.1754918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:39:38.1756289Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:38.1757127Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:38.1758225Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:38.1759094Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:38.1759910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:39:38.1761155Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:39:38.1762470Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:39:38.1763546Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:39:38.1764521Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:39:38.1765358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:39:38.1766223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:39:38.1766879Z dist init r=0, world=2 2022-11-23T03:39:38.1767336Z dist init r=1, world=2 2022-11-23T03:39:38.1772799Z ok (6.635s) 2022-11-23T03:39:38.1773411Z test_ignored_modules_invalid (__main__.TestFSDPIgnoredModules) 2022-11-23T03:39:38.1774377Z Tests that passing an FSDP module as an ignored module or the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39216 2022-11-23T03:39:38.1775357Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39217 2022-11-23T03:39:38.1776580Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:38.1777425Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:38.1778529Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:38.1779390Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:38.1780205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:39:38.1781497Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:38.1782321Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:38.1783428Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:38.1784281Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:38.1785111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:39:38.1786529Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:39:38.1787836Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:39:38.1788899Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:39:38.1789852Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:39:38.1790686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:39:38.1791557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:39:38.1792197Z dist init r=1, world=2 2022-11-23T03:39:38.1792654Z dist init r=0, world=2 2022-11-23T03:39:38.1793093Z ok (4.928s) 2022-11-23T03:39:38.1793779Z test_ignored_modules_nested (__main__.TestFSDPIgnoredModules) 2022-11-23T03:39:38.1794751Z Tests that passing a module with nested FSDP modules does not ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39359 2022-11-23T03:39:38.1795729Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39360 2022-11-23T03:39:38.1796892Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:38.1797717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:38.1798818Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:38.1799680Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:38.1800492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:39:38.1801690Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:38.1802517Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:38.1803602Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:38.1804465Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:38.1805282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:39:38.1806526Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:39:38.1807950Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:39:38.1809029Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:39:38.1809978Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:39:38.1810851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:39:38.1811728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:39:38.1812409Z dist init r=1, world=2 2022-11-23T03:39:38.1812874Z dist init r=0, world=2 2022-11-23T03:39:38.1813318Z ok (6.738s) 2022-11-23T03:39:38.1813921Z test_ignored_modules_transformer (__main__.TestFSDPIgnoredModules) 2022-11-23T03:39:38.1815217Z Tests that ignored modules' parameters are not flattened for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39512 2022-11-23T03:39:38.1816217Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39513 2022-11-23T03:39:38.1817387Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:38.1818213Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:38.1819477Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:38.1820343Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:38.1821163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:39:38.1822350Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:38.1823170Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:38.1824267Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:38.1825122Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:38.1825947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:39:38.1827292Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:39:38.1828616Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:39:38.1829694Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:39:38.1830652Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:39:38.1831482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:39:38.1832360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:39:38.1833002Z dist init r=0, world=2 2022-11-23T03:39:38.1833463Z dist init r=1, world=2 2022-11-23T03:39:38.1833903Z ok (6.936s) 2022-11-23T03:39:38.1834159Z 2022-11-23T03:39:38.1834683Z ---------------------------------------------------------------------- 2022-11-23T03:39:38.1835279Z Ran 5 tests in 32.460s 2022-11-23T03:39:38.1835578Z 2022-11-23T03:39:38.1835733Z OK 2022-11-23T03:39:38.1835964Z 2022-11-23T03:39:38.1836177Z Generating XML reports... 2022-11-23T03:39:38.1837387Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules/TEST-TestFSDPIgnoredModules-20221123033903.xml 2022-11-23T03:39:38.1838062Z 2022-11-23T03:39:38.1838662Z ##[endgroup] 2022-11-23T03:39:38.1839910Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_ignored_modules (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_ignored_modules_2jqpb84t) 2022-11-23T03:39:38.1840601Z 2022-11-23T03:39:38.1841092Z Running distributed/fsdp/test_fsdp_fx ... [2022-11-23 03:39:38.171511] 2022-11-23T03:39:38.1842363Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_fx.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:39:38.172188] 2022-11-23T03:39:47.3032051Z 2022-11-23T03:39:47.3032848Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_fx 2022-11-23T03:39:47.3035734Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_fx (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_fx_524uo4wf) 2022-11-23T03:39:47.3038256Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_fx 2022-11-23T03:39:47.3039107Z 2022-11-23T03:39:47.3039434Z Running tests... 2022-11-23T03:39:47.3040731Z ---------------------------------------------------------------------- 2022-11-23T03:39:47.3041760Z test_symbolic_tracing_outputs (__main__.TestSymbolicTracing) 2022-11-23T03:39:47.3043308Z test ``execution_info.module_forward_order`` and ``execution_info.module_to_execution_infos`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39732 2022-11-23T03:39:47.3045070Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39733 2022-11-23T03:39:47.3047015Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:47.3048817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:47.3050480Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:47.3051690Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:47.3052822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:39:47.3054489Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:47.3055624Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:47.3057121Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:47.3058314Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:47.3059678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:39:47.3061423Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:39:47.3063229Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:39:47.3064708Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:39:47.3066026Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:39:47.3067171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:39:47.3068352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:39:47.3069245Z dist init r=1, world=2 2022-11-23T03:39:47.3069877Z dist init r=0, world=2 2022-11-23T03:39:47.3070484Z ok (5.123s) 2022-11-23T03:39:47.3070853Z 2022-11-23T03:39:47.3071574Z ---------------------------------------------------------------------- 2022-11-23T03:39:47.3072393Z Ran 1 test in 5.124s 2022-11-23T03:39:47.3072783Z 2022-11-23T03:39:47.3072978Z OK 2022-11-23T03:39:47.3073297Z 2022-11-23T03:39:47.3073588Z Generating XML reports... 2022-11-23T03:39:47.3075161Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_fx/TEST-TestSymbolicTracing-20221123033940.xml 2022-11-23T03:39:47.3076035Z 2022-11-23T03:39:47.3076775Z ##[endgroup] 2022-11-23T03:39:47.3078326Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_fx (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_fx_524uo4wf) 2022-11-23T03:39:47.3079171Z 2022-11-23T03:39:47.3079922Z Running distributed/fsdp/test_fsdp_flatten_params ... [2022-11-23 03:39:47.303424] 2022-11-23T03:39:47.3081826Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_flatten_params.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:39:47.304091] 2022-11-23T03:40:38.1125182Z 2022-11-23T03:40:38.1128559Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_flatten_params 2022-11-23T03:40:38.1131234Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_flatten_params (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_flatten_params_cvh1auw0) 2022-11-23T03:40:38.1133459Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_flatten_params 2022-11-23T03:40:38.1134077Z 2022-11-23T03:40:38.1136067Z Running tests... 2022-11-23T03:40:38.1136962Z ---------------------------------------------------------------------- 2022-11-23T03:40:38.1137806Z test_empty_module (__main__.TestFlattenParams) 2022-11-23T03:40:38.1138875Z Tests flattening an empty module (i.e. one without any parameters). ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39942 2022-11-23T03:40:38.1140793Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:40:38.1142789Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:40:38.1144311Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:40:38.1145587Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:40:38.1146486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:40:38.1148021Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:40:38.1149602Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:40:38.1151018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:40:38.1151850Z dist init r=0, world=1 2022-11-23T03:40:38.1152368Z ok (4.906s) 2022-11-23T03:40:38.1153495Z test_flat_param_shard_metadata (__main__.TestFlattenParams) 2022-11-23T03:40:38.1154722Z Tests that ``FlatParameter`` shard metadata are computed as expected. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40014 2022-11-23T03:40:38.1156549Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:40:38.1157598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:40:38.1158862Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:40:38.1190893Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:40:38.1192815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:40:38.1195238Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:40:38.1243517Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:40:38.1245350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:40:38.1246438Z dist init r=0, world=1 2022-11-23T03:40:38.1247103Z ok (4.725s) 2022-11-23T03:40:38.1248057Z test_flatten_nothing (__main__.TestFlattenParams) 2022-11-23T03:40:38.1249500Z Tests that constructing a ``FlatParamHandle`` with no parameters ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40086 2022-11-23T03:40:38.1251513Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:40:38.1252711Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:40:38.1254386Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:40:38.1255695Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:40:38.1256936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:40:38.1258768Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:40:38.1260367Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:40:38.1261574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:40:38.1262504Z dist init r=0, world=1 2022-11-23T03:40:38.1263166Z ok (4.319s) 2022-11-23T03:40:38.1264046Z test_numel_with_shared_params (__main__.TestFlattenParams) 2022-11-23T03:40:38.1265512Z Tests that numel is preserved after flattening when there are shared ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40158 2022-11-23T03:40:38.1267521Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:40:38.1269358Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:40:38.1271049Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:40:38.1272308Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:40:38.1273505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:40:38.1275331Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:40:38.1276907Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:40:38.1278098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:40:38.1279075Z dist init r=0, world=1 2022-11-23T03:40:38.1279704Z ok (4.319s) 2022-11-23T03:40:38.1280515Z test_numel_without_shared_params (__main__.TestFlattenParams) 2022-11-23T03:40:38.1282166Z Tests that numel is preserved after flattening when there are no shared ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40230 2022-11-23T03:40:38.1284271Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:40:38.1285503Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:40:38.1287172Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:40:38.1288805Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:40:38.1290066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:40:38.1291961Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:40:38.1293602Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:40:38.1294884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:40:38.1295779Z dist init r=0, world=1 2022-11-23T03:40:38.1296288Z ok (4.419s) 2022-11-23T03:40:38.1296903Z test_output_with_shared_params (__main__.TestFlattenParams) 2022-11-23T03:40:38.1297910Z Tests a forward pass after flattening when there are shared parameters ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40302 2022-11-23T03:40:38.1299300Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:40:38.1300161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:40:38.1301283Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:40:38.1302191Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:40:38.1303063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:40:38.1304350Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:40:38.1305472Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:40:38.1306340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:40:38.1307035Z dist init r=0, world=1 2022-11-23T03:40:38.1307489Z ok (6.427s) 2022-11-23T03:40:38.1308099Z test_output_without_shared_params (__main__.TestFlattenParams) 2022-11-23T03:40:38.1309087Z Tests a forward pass after flattening when there are no shared ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40377 2022-11-23T03:40:38.1310433Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:40:38.1311452Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:40:38.1312607Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:40:38.1313515Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:40:38.1314338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:40:38.1315611Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:40:38.1316721Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:40:38.1317592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:40:38.1318285Z dist init r=0, world=1 2022-11-23T03:40:38.1318761Z ok (6.523s) 2022-11-23T03:40:38.1319353Z test_partial_flattening (__main__.TestFlattenParams) 2022-11-23T03:40:38.1320364Z Tests flattening some submodules but not others. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40452 2022-11-23T03:40:38.1321713Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:40:38.1322570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:40:38.1323721Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:40:38.1324631Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:40:38.1325492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:40:38.1326785Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:40:38.1327961Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:40:38.1328860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:40:38.1329562Z dist init r=0, world=1 2022-11-23T03:40:38.1332011Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:40:38.1333541Z warnings.warn( 2022-11-23T03:40:38.1334024Z ok (4.620s) 2022-11-23T03:40:38.1334657Z test_pnorm_after_step_with_shared_params (__main__.TestFlattenParams) 2022-11-23T03:40:38.1335690Z Tests for parameter Frobenius norm parity after an optimizer step when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40524 2022-11-23T03:40:38.1337105Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:40:38.1337980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:40:38.1339087Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:40:38.1339994Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:40:38.1340847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:40:38.1342129Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:40:38.1343254Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:40:38.1344136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:40:38.1344977Z dist init r=0, world=1 2022-11-23T03:40:38.1345421Z ok (6.622s) 2022-11-23T03:40:38.1345693Z 2022-11-23T03:40:38.1346250Z ---------------------------------------------------------------------- 2022-11-23T03:40:38.1346899Z Ran 9 tests in 46.883s 2022-11-23T03:40:38.1347205Z 2022-11-23T03:40:38.1347376Z OK 2022-11-23T03:40:38.1347634Z 2022-11-23T03:40:38.1347869Z Generating XML reports... 2022-11-23T03:40:38.1349088Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_flatten_params/TEST-TestFlattenParams-20221123033949.xml 2022-11-23T03:40:38.1349755Z 2022-11-23T03:40:38.1350444Z ##[endgroup] 2022-11-23T03:40:38.1351719Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_flatten_params (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_flatten_params_cvh1auw0) 2022-11-23T03:40:38.1352412Z 2022-11-23T03:40:38.1352909Z Running distributed/fsdp/test_fsdp_core ... [2022-11-23 03:40:38.113308] 2022-11-23T03:40:38.1354308Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_core.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:40:38.114231] 2022-11-23T03:54:45.8940955Z 2022-11-23T03:54:45.8944620Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_core 2022-11-23T03:54:45.8947099Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_core (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_core_90guj51q) 2022-11-23T03:54:45.8993603Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_core 2022-11-23T03:54:45.8994373Z 2022-11-23T03:54:45.8994622Z Running tests... 2022-11-23T03:54:45.8999471Z ---------------------------------------------------------------------- 2022-11-23T03:54:45.9000741Z test_pre_backward_hook_registration_after_state_dict (__main__.TestHooks) 2022-11-23T03:54:45.9003313Z Tests that FSDP pre-backward hooks are registered on forward pass ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40668 2022-11-23T03:54:45.9005123Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40669 2022-11-23T03:54:45.9007312Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:45.9009573Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:45.9011638Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:45.9012963Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:45.9014692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:45.9016835Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:45.9018036Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:45.9019629Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:45.9021054Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:45.9022581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:45.9024649Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:45.9026667Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:45.9028899Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:45.9030659Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:45.9032059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:45.9033565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:45.9034967Z dist init r=0, world=2 2022-11-23T03:54:45.9039364Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:45.9042071Z warnings.warn( 2022-11-23T03:54:45.9042968Z dist init r=1, world=2 2022-11-23T03:54:45.9047682Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:45.9051335Z warnings.warn( 2022-11-23T03:54:45.9052139Z ok (7.841s) 2022-11-23T03:54:45.9053335Z test_pre_backward_hook_registration_cuda_first_False (__main__.TestHooks) 2022-11-23T03:54:45.9055845Z Tests that FSDP pre-backward hooks are registered on forward pass ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40821 2022-11-23T03:54:45.9057893Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40822 2022-11-23T03:54:45.9059698Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:45.9060870Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:45.9062412Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:45.9063828Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:45.9064992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:45.9066816Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:45.9069535Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:45.9071025Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:45.9072146Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:45.9073260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:45.9074935Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:45.9076810Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:45.9078365Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:45.9079818Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:45.9081005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:45.9082218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:45.9083141Z dist init r=1, world=2 2022-11-23T03:54:45.9086271Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:45.9088542Z warnings.warn( 2022-11-23T03:54:45.9089141Z dist init r=0, world=2 2022-11-23T03:54:45.9092247Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:45.9094247Z warnings.warn( 2022-11-23T03:54:45.9094873Z ok (6.733s) 2022-11-23T03:54:45.9095687Z test_pre_backward_hook_registration_cuda_first_True (__main__.TestHooks) 2022-11-23T03:54:45.9097696Z Tests that FSDP pre-backward hooks are registered on forward pass ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40974 2022-11-23T03:54:45.9098998Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40975 2022-11-23T03:54:45.9100070Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:45.9100751Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:45.9101734Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:45.9102477Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:45.9103188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:45.9104252Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:45.9105012Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:45.9106143Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:45.9106863Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:45.9107620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:45.9108674Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:45.9109867Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:45.9110880Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:45.9111679Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:45.9112424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:45.9113187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:45.9113860Z dist init r=1, world=2 2022-11-23T03:54:45.9114294Z dist init r=0, world=2 2022-11-23T03:54:45.9114780Z ok (6.835s) 2022-11-23T03:54:45.9115292Z test_register_functions_called_cuda_first_False_mixed_precision_False (__main__.TestHooks) 2022-11-23T03:54:45.9116249Z Tests that ``_register_{pre|post}_backward_hooks()`` are called ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41127 2022-11-23T03:54:45.9117107Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41128 2022-11-23T03:54:45.9118306Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:45.9119061Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:45.9120052Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:45.9120982Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:45.9121710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:45.9122705Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:45.9123586Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:45.9124761Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:45.9125591Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:45.9126391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:45.9127598Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:45.9129025Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:45.9130129Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:45.9130964Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:45.9131691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:45.9132404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:45.9133089Z dist init r=0, world=2 2022-11-23T03:54:45.9135184Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:45.9136565Z warnings.warn( 2022-11-23T03:54:45.9136983Z dist init r=1, world=2 2022-11-23T03:54:45.9138963Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:45.9140202Z warnings.warn( 2022-11-23T03:54:45.9140685Z ok (6.536s) 2022-11-23T03:54:45.9141353Z test_register_functions_called_cuda_first_False_mixed_precision_True (__main__.TestHooks) 2022-11-23T03:54:45.9142314Z Tests that ``_register_{pre|post}_backward_hooks()`` are called ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41276 2022-11-23T03:54:45.9143143Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41277 2022-11-23T03:54:45.9144173Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:45.9145015Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:45.9145984Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:45.9146737Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:45.9147424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:45.9148449Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:45.9149131Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:45.9150030Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:45.9150894Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:45.9151586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:45.9152689Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:45.9153787Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:45.9154825Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:45.9155627Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:45.9156400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:45.9157346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:45.9157959Z dist init r=1, world=2 2022-11-23T03:54:45.9159789Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_wrap_utils.py:68: UserWarning: Both mixed precision and an `auto_wrap_policy` were specified for FSDP, where the wrapped module has batch norm submodules. The batch norm submodules will be wrapped as separate FSDP instances with mixed precision disabled since some batch norm kernels do not support low precision. 2022-11-23T03:54:45.9160876Z warnings.warn( 2022-11-23T03:54:45.9163029Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:45.9164385Z warnings.warn( 2022-11-23T03:54:45.9164794Z dist init r=0, world=2 2022-11-23T03:54:45.9166468Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_wrap_utils.py:68: UserWarning: Both mixed precision and an `auto_wrap_policy` were specified for FSDP, where the wrapped module has batch norm submodules. The batch norm submodules will be wrapped as separate FSDP instances with mixed precision disabled since some batch norm kernels do not support low precision. 2022-11-23T03:54:45.9167610Z warnings.warn( 2022-11-23T03:54:45.9169904Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:45.9171172Z warnings.warn( 2022-11-23T03:54:45.9171541Z ok (7.040s) 2022-11-23T03:54:45.9172098Z test_register_functions_called_cuda_first_True_mixed_precision_False (__main__.TestHooks) 2022-11-23T03:54:45.9172941Z Tests that ``_register_{pre|post}_backward_hooks()`` are called ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41425 2022-11-23T03:54:45.9173739Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41426 2022-11-23T03:54:45.9174745Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:45.9175587Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:45.9176495Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:45.9177267Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:45.9178388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:45.9179469Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:45.9180179Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:45.9181165Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:45.9181930Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:45.9182623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:45.9183669Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:45.9185051Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:45.9186011Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:45.9186863Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:45.9187573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:45.9188352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:45.9188899Z dist init r=1, world=2 2022-11-23T03:54:45.9189260Z dist init r=0, world=2 2022-11-23T03:54:45.9189701Z ok (6.636s) 2022-11-23T03:54:45.9190296Z test_register_functions_called_cuda_first_True_mixed_precision_True (__main__.TestHooks) 2022-11-23T03:54:45.9191214Z Tests that ``_register_{pre|post}_backward_hooks()`` are called ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41574 2022-11-23T03:54:45.9192076Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41575 2022-11-23T03:54:45.9193183Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:45.9194029Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:45.9195194Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:45.9196010Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:45.9196822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:45.9197824Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:45.9198544Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:45.9199650Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:45.9200533Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:45.9201237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:45.9202317Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:45.9203517Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:45.9204491Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:45.9205300Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:45.9205996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:45.9206791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:45.9207377Z dist init r=0, world=2 2022-11-23T03:54:45.9207946Z dist init r=1, world=2 2022-11-23T03:54:45.9209886Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_wrap_utils.py:68: UserWarning: Both mixed precision and an `auto_wrap_policy` were specified for FSDP, where the wrapped module has batch norm submodules. The batch norm submodules will be wrapped as separate FSDP instances with mixed precision disabled since some batch norm kernels do not support low precision. 2022-11-23T03:54:45.9211051Z warnings.warn( 2022-11-23T03:54:45.9212842Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_wrap_utils.py:68: UserWarning: Both mixed precision and an `auto_wrap_policy` were specified for FSDP, where the wrapped module has batch norm submodules. The batch norm submodules will be wrapped as separate FSDP instances with mixed precision disabled since some batch norm kernels do not support low precision. 2022-11-23T03:54:45.9213967Z warnings.warn( 2022-11-23T03:54:45.9214379Z ok (6.634s) 2022-11-23T03:54:45.9215045Z test_transformer_no_grad_mixed_precision_False (__main__.TestNoGrad) 2022-11-23T03:54:45.9216164Z Tests that for an FSDP-wrapped transformer model with shared ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41723 2022-11-23T03:54:45.9217064Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41724 2022-11-23T03:54:45.9218241Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:45.9219085Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:45.9219953Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:45.9220773Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:45.9221504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:45.9222522Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:45.9223313Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:45.9224286Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:45.9225118Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:45.9225823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:45.9226897Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:45.9228006Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:45.9228980Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:45.9229962Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:45.9230691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:45.9231495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:45.9232092Z dist init r=1, world=2 2022-11-23T03:54:45.9234332Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:45.9235566Z warnings.warn( 2022-11-23T03:54:45.9236017Z dist init r=0, world=2 2022-11-23T03:54:45.9238138Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:45.9239455Z warnings.warn( 2022-11-23T03:54:45.9239865Z ok (6.838s) 2022-11-23T03:54:45.9240423Z test_transformer_no_grad_mixed_precision_True (__main__.TestNoGrad) 2022-11-23T03:54:45.9241590Z Tests that for an FSDP-wrapped transformer model with shared ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41876 2022-11-23T03:54:45.9242548Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41877 2022-11-23T03:54:45.9243713Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:45.9244497Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:45.9245481Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:45.9246207Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:45.9247149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:45.9248248Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:45.9249018Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:45.9250135Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:45.9250881Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:45.9251596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:45.9252655Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:45.9253773Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:45.9254764Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:45.9255684Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:45.9256460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:45.9257323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:45.9257903Z dist init r=1, world=2 2022-11-23T03:54:45.9259778Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_wrap_utils.py:68: UserWarning: Both mixed precision and an `auto_wrap_policy` were specified for FSDP, where the wrapped module has batch norm submodules. The batch norm submodules will be wrapped as separate FSDP instances with mixed precision disabled since some batch norm kernels do not support low precision. 2022-11-23T03:54:45.9260945Z warnings.warn( 2022-11-23T03:54:45.9262913Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:45.9264134Z warnings.warn( 2022-11-23T03:54:45.9264540Z dist init r=0, world=2 2022-11-23T03:54:45.9266247Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_wrap_utils.py:68: UserWarning: Both mixed precision and an `auto_wrap_policy` were specified for FSDP, where the wrapped module has batch norm submodules. The batch norm submodules will be wrapped as separate FSDP instances with mixed precision disabled since some batch norm kernels do not support low precision. 2022-11-23T03:54:45.9267557Z warnings.warn( 2022-11-23T03:54:45.9269617Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:45.9270903Z warnings.warn( 2022-11-23T03:54:45.9271310Z ok (7.235s) 2022-11-23T03:54:45.9271820Z test_param_change_after_init_mixed_precision_False (__main__.TestParamInit) 2022-11-23T03:54:45.9273049Z Tests that changing FSDP model parameter values in-place after FSDP ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42029 2022-11-23T03:54:45.9274020Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42030 2022-11-23T03:54:45.9275066Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:45.9275829Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:45.9276844Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:45.9277664Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:45.9278403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:45.9279514Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:45.9280248Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:45.9281377Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:45.9282114Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:45.9282830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:45.9283942Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:45.9285198Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:45.9286160Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:45.9287043Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:45.9288004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:45.9288788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:45.9289410Z dist init r=1, world=2 2022-11-23T03:54:45.9291523Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:45.9292809Z warnings.warn( 2022-11-23T03:54:45.9293247Z dist init r=0, world=2 2022-11-23T03:54:45.9295298Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:45.9296878Z warnings.warn( 2022-11-23T03:54:45.9297324Z ok (6.735s) 2022-11-23T03:54:45.9297825Z test_param_change_after_init_mixed_precision_True (__main__.TestParamInit) 2022-11-23T03:54:45.9299051Z Tests that changing FSDP model parameter values in-place after FSDP ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42178 2022-11-23T03:54:45.9300055Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42179 2022-11-23T03:54:45.9301164Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:45.9301895Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:45.9303020Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:45.9303808Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:45.9304582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:45.9305591Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:45.9306292Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:45.9307237Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:45.9307911Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:45.9308580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:45.9309684Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:45.9310845Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:45.9311847Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:45.9312779Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:45.9313542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:45.9314350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:45.9314918Z dist init r=1, world=2 2022-11-23T03:54:45.9315362Z dist init r=0, world=2 2022-11-23T03:54:45.9317166Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_wrap_utils.py:68: UserWarning: Both mixed precision and an `auto_wrap_policy` were specified for FSDP, where the wrapped module has batch norm submodules. The batch norm submodules will be wrapped as separate FSDP instances with mixed precision disabled since some batch norm kernels do not support low precision. 2022-11-23T03:54:45.9318247Z warnings.warn( 2022-11-23T03:54:45.9320266Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:45.9321610Z warnings.warn( 2022-11-23T03:54:45.9323347Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_wrap_utils.py:68: UserWarning: Both mixed precision and an `auto_wrap_policy` were specified for FSDP, where the wrapped module has batch norm submodules. The batch norm submodules will be wrapped as separate FSDP instances with mixed precision disabled since some batch norm kernels do not support low precision. 2022-11-23T03:54:45.9324634Z warnings.warn( 2022-11-23T03:54:45.9326658Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:45.9327915Z warnings.warn( 2022-11-23T03:54:45.9328278Z ok (7.537s) 2022-11-23T03:54:45.9328899Z test_delayed_optim_step_offload_false_no_shard (__main__.TestParityWithDDP) 2022-11-23T03:54:45.9329930Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42327 2022-11-23T03:54:45.9330887Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42328 2022-11-23T03:54:45.9332081Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:45.9332835Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:45.9333935Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:45.9334653Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:45.9335373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:45.9336549Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:45.9337235Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:45.9338333Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:45.9339151Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:45.9339897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:45.9341080Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:45.9342247Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:45.9343298Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:45.9344216Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:45.9344989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:45.9345890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:45.9346792Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9347578Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9349845Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:45.9351149Z warnings.warn( 2022-11-23T03:54:45.9351759Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9354038Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:45.9355536Z warnings.warn( 2022-11-23T03:54:45.9356085Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9356849Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9357682Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9358527Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9359360Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9360164Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9360948Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9361658Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9362494Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9363348Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9364169Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9364932Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9365739Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9366649Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9367380Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9368441Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9369193Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9370064Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9370823Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9372496Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:45.9374582Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:45.9376725Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:45.9378900Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:45.9381010Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:45.9383273Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:45.9385450Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:45.9387748Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:45.9389939Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:45.9391977Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:45.9393143Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9393981Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9394809Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9395629Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9396362Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9397263Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9398062Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9398885Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9399685Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9400458Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9401403Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9402207Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9402934Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9403681Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9404442Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9405305Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9406241Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9407046Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9408079Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9408963Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9409688Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9410442Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9411228Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9411989Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9413892Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:45.9415987Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:45.9418209Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:45.9420337Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:45.9422431Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:45.9424523Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:45.9426643Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:45.9428759Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:45.9430056Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9430875Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9431428Z dist init r=1, world=2 2022-11-23T03:54:45.9431854Z dist init r=0, world=2 2022-11-23T03:54:45.9432256Z ok (36.487s) 2022-11-23T03:54:45.9432817Z test_delayed_optim_step_offload_false_none (__main__.TestParityWithDDP) 2022-11-23T03:54:45.9433714Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42480 2022-11-23T03:54:45.9434706Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42481 2022-11-23T03:54:45.9435765Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:45.9436529Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:45.9437553Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:45.9438354Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:45.9439029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:45.9440120Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:45.9440971Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:45.9442081Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:45.9442862Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:45.9443600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:45.9444743Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:45.9446002Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:45.9446955Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:45.9447875Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:45.9448552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:45.9449315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:45.9450290Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9451083Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9453280Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:45.9454677Z warnings.warn( 2022-11-23T03:54:45.9455281Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9457539Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:45.9458895Z warnings.warn( 2022-11-23T03:54:45.9459439Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9460244Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9460993Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9461797Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9462511Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9463364Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9464131Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9465033Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9465785Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9466550Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9467281Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9468065Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9468781Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9469664Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9470537Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9471303Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9472294Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9473124Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9473883Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9474613Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9475312Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9476118Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9476941Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9477677Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9478495Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9479211Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9479948Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9480737Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9481500Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9482204Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9482961Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9483701Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9484408Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9485238Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9486010Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9486733Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9487514Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9488413Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9489158Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9489923Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9490856Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9491558Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9492378Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9493172Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9494043Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9494697Z dist init r=0, world=2 2022-11-23T03:54:45.9495137Z dist init r=1, world=2 2022-11-23T03:54:45.9495515Z ok (40.991s) 2022-11-23T03:54:45.9496137Z test_delayed_optim_step_offload_false_shard_grad_op (__main__.TestParityWithDDP) 2022-11-23T03:54:45.9497005Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42633 2022-11-23T03:54:45.9497903Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42634 2022-11-23T03:54:45.9499142Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:45.9499811Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:45.9500823Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:45.9501634Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:45.9502407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:45.9503482Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:45.9504263Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:45.9505295Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:45.9506028Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:45.9506836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:45.9507872Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:45.9509091Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:45.9510098Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:45.9510988Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:45.9511752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:45.9512596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:45.9513448Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9514243Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9516579Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:45.9517842Z warnings.warn( 2022-11-23T03:54:45.9518495Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9520544Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:45.9521876Z warnings.warn( 2022-11-23T03:54:45.9522500Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9523245Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9523962Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9524705Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9525382Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9526300Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9527054Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9527958Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9528668Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9529358Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9530083Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9530771Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9531662Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9532448Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9533284Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9534078Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9534884Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9535686Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9536377Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9537095Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9537864Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9538610Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9539548Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9540446Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9541191Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9541981Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9542722Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9543624Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9544345Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9545162Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9545940Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9546880Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9547621Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9548332Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9549129Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9549883Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9550730Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9551546Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9552288Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9553066Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9554089Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9554910Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9555765Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9556680Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9557509Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9558117Z dist init r=1, world=2 2022-11-23T03:54:45.9558531Z dist init r=0, world=2 2022-11-23T03:54:45.9558930Z ok (33.778s) 2022-11-23T03:54:45.9559470Z test_delayed_optim_step_offload_true_no_shard (__main__.TestParityWithDDP) 2022-11-23T03:54:45.9561446Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82490 for platform(s) linux, rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.003s) 2022-11-23T03:54:45.9562676Z test_delayed_optim_step_offload_true_none (__main__.TestParityWithDDP) 2022-11-23T03:54:45.9563565Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42786 2022-11-23T03:54:45.9564386Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42787 2022-11-23T03:54:45.9565418Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:45.9566126Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:45.9567090Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:45.9568008Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:45.9568689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:45.9569756Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:45.9570448Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:45.9571415Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:45.9572169Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:45.9572952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:45.9573959Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:45.9575033Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:45.9576052Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:45.9576862Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:45.9577719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:45.9578494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:45.9579219Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9580037Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9582312Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:45.9583554Z warnings.warn( 2022-11-23T03:54:45.9583976Z File "", line 1, in 2022-11-23T03:54:45.9584572Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:45.9585209Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:45.9585788Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:45.9586414Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:45.9587128Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:45.9587657Z self.run() 2022-11-23T03:54:45.9588187Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:45.9588789Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:45.9589713Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:45.9590326Z self.run_test(test_name, pipe) 2022-11-23T03:54:45.9591216Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:45.9592004Z getattr(self, test_name)() 2022-11-23T03:54:45.9592939Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:45.9593525Z fn() 2022-11-23T03:54:45.9594428Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:45.9595066Z test(self, **param_kwargs) 2022-11-23T03:54:45.9595925Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:45.9596566Z return func(*args, **kwargs) 2022-11-23T03:54:45.9597364Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:45.9597912Z self.run_subtests( 2022-11-23T03:54:45.9598740Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:45.9599398Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:45.9600445Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:45.9601222Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:45.9602174Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:45.9602821Z output = model(*input) 2022-11-23T03:54:45.9603648Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:45.9604417Z return forward_call(*input, **kwargs) 2022-11-23T03:54:45.9605376Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:45.9606056Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:45.9607055Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:45.9607818Z _lazy_init(state, module) 2022-11-23T03:54:45.9608784Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:45.9609424Z handle.init_flat_param_attributes() 2022-11-23T03:54:45.9610485Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:45.9611145Z return func(*args, **kwargs) 2022-11-23T03:54:45.9612204Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:45.9612853Z p_assert( 2022-11-23T03:54:45.9613714Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:45.9614380Z traceback.print_stack() 2022-11-23T03:54:45.9615042Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9617335Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:45.9618614Z warnings.warn( 2022-11-23T03:54:45.9619061Z File "", line 1, in 2022-11-23T03:54:45.9619710Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:45.9620309Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:45.9620923Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:45.9621500Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:45.9622175Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:45.9622698Z self.run() 2022-11-23T03:54:45.9623231Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:45.9623783Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:45.9624720Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:45.9625337Z self.run_test(test_name, pipe) 2022-11-23T03:54:45.9626155Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:45.9626827Z getattr(self, test_name)() 2022-11-23T03:54:45.9627679Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:45.9628338Z fn() 2022-11-23T03:54:45.9629217Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:45.9629911Z test(self, **param_kwargs) 2022-11-23T03:54:45.9630806Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:45.9631496Z return func(*args, **kwargs) 2022-11-23T03:54:45.9632204Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:45.9632807Z self.run_subtests( 2022-11-23T03:54:45.9633676Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:45.9634589Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:45.9635588Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:45.9636313Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:45.9637199Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:45.9637946Z output = model(*input) 2022-11-23T03:54:45.9638735Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:45.9639380Z return forward_call(*input, **kwargs) 2022-11-23T03:54:45.9640375Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:45.9641158Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:45.9642190Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:45.9642800Z _lazy_init(state, module) 2022-11-23T03:54:45.9643625Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:45.9644249Z handle.init_flat_param_attributes() 2022-11-23T03:54:45.9645120Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:45.9645714Z return func(*args, **kwargs) 2022-11-23T03:54:45.9646632Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:45.9647231Z p_assert( 2022-11-23T03:54:45.9648062Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:45.9648638Z traceback.print_stack() 2022-11-23T03:54:45.9649284Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9649953Z File "", line 1, in 2022-11-23T03:54:45.9650521Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:45.9651113Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:45.9651718Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:45.9652287Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:45.9652936Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:45.9653519Z self.run() 2022-11-23T03:54:45.9654102Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:45.9654749Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:45.9655721Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:45.9656321Z self.run_test(test_name, pipe) 2022-11-23T03:54:45.9657296Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:45.9658006Z getattr(self, test_name)() 2022-11-23T03:54:45.9658849Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:45.9659541Z fn() 2022-11-23T03:54:45.9660439Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:45.9661055Z test(self, **param_kwargs) 2022-11-23T03:54:45.9662025Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:45.9662623Z return func(*args, **kwargs) 2022-11-23T03:54:45.9663300Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:45.9663919Z self.run_subtests( 2022-11-23T03:54:45.9664787Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:45.9665597Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:45.9666478Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:45.9667154Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:45.9668132Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:45.9668760Z output = model(*input) 2022-11-23T03:54:45.9669651Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:45.9670281Z return forward_call(*input, **kwargs) 2022-11-23T03:54:45.9671297Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:45.9672080Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:45.9673162Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:45.9673794Z _lazy_init(state, module) 2022-11-23T03:54:45.9674641Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:45.9675340Z handle.init_flat_param_attributes() 2022-11-23T03:54:45.9676218Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:45.9676772Z return func(*args, **kwargs) 2022-11-23T03:54:45.9677650Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:45.9678222Z p_assert( 2022-11-23T03:54:45.9678989Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:45.9679577Z traceback.print_stack() 2022-11-23T03:54:45.9680244Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9680828Z File "", line 1, in 2022-11-23T03:54:45.9681479Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:45.9682014Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:45.9682678Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:45.9683327Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:45.9684010Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:45.9684539Z self.run() 2022-11-23T03:54:45.9685144Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:45.9685695Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:45.9686542Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:45.9687166Z self.run_test(test_name, pipe) 2022-11-23T03:54:45.9688313Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:45.9688959Z getattr(self, test_name)() 2022-11-23T03:54:45.9689831Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:45.9690428Z fn() 2022-11-23T03:54:45.9691240Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:45.9691843Z test(self, **param_kwargs) 2022-11-23T03:54:45.9692737Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:45.9693402Z return func(*args, **kwargs) 2022-11-23T03:54:45.9694143Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:45.9694933Z self.run_subtests( 2022-11-23T03:54:45.9695769Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:45.9696382Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:45.9697315Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:45.9697987Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:45.9698900Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:45.9699560Z output = model(*input) 2022-11-23T03:54:45.9700406Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:45.9701002Z return forward_call(*input, **kwargs) 2022-11-23T03:54:45.9702027Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:45.9702786Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:45.9703818Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:45.9704455Z _lazy_init(state, module) 2022-11-23T03:54:45.9705349Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:45.9705975Z handle.init_flat_param_attributes() 2022-11-23T03:54:45.9706917Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:45.9707500Z return func(*args, **kwargs) 2022-11-23T03:54:45.9708492Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:45.9709102Z p_assert( 2022-11-23T03:54:45.9709892Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:45.9710534Z traceback.print_stack() 2022-11-23T03:54:45.9711155Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9711731Z File "", line 1, in 2022-11-23T03:54:45.9712296Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:45.9712969Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:45.9713560Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:45.9714155Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:45.9714764Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:45.9715304Z self.run() 2022-11-23T03:54:45.9715810Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:45.9716409Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:45.9717343Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:45.9718085Z self.run_test(test_name, pipe) 2022-11-23T03:54:45.9719020Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:45.9719648Z getattr(self, test_name)() 2022-11-23T03:54:45.9720509Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:45.9721128Z fn() 2022-11-23T03:54:45.9722025Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:45.9722647Z test(self, **param_kwargs) 2022-11-23T03:54:45.9723530Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:45.9724108Z return func(*args, **kwargs) 2022-11-23T03:54:45.9724895Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:45.9725461Z self.run_subtests( 2022-11-23T03:54:45.9726279Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:45.9726905Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:45.9727687Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:45.9728351Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:45.9729252Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:45.9729968Z output = model(*input) 2022-11-23T03:54:45.9730900Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:45.9731661Z return forward_call(*input, **kwargs) 2022-11-23T03:54:45.9732746Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:45.9733604Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:45.9734743Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:45.9735352Z _lazy_init(state, module) 2022-11-23T03:54:45.9736422Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:45.9737053Z handle.init_flat_param_attributes() 2022-11-23T03:54:45.9738120Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:45.9738734Z return func(*args, **kwargs) 2022-11-23T03:54:45.9739758Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:45.9740386Z p_assert( 2022-11-23T03:54:45.9741201Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:45.9741884Z traceback.print_stack() 2022-11-23T03:54:45.9742440Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9742978Z File "", line 1, in 2022-11-23T03:54:45.9743456Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:45.9743940Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:45.9744404Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:45.9744887Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:45.9745394Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:45.9745835Z self.run() 2022-11-23T03:54:45.9746276Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:45.9746753Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:45.9747432Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:45.9747933Z self.run_test(test_name, pipe) 2022-11-23T03:54:45.9748648Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:45.9749162Z getattr(self, test_name)() 2022-11-23T03:54:45.9749848Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:45.9750327Z fn() 2022-11-23T03:54:45.9750995Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:45.9751493Z test(self, **param_kwargs) 2022-11-23T03:54:45.9752196Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:45.9752814Z return func(*args, **kwargs) 2022-11-23T03:54:45.9753339Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:45.9753828Z self.run_subtests( 2022-11-23T03:54:45.9754510Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:45.9755039Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:45.9755776Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:45.9756324Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:45.9757075Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:45.9757598Z output = model(*input) 2022-11-23T03:54:45.9758306Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:45.9758826Z return forward_call(*input, **kwargs) 2022-11-23T03:54:45.9759545Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:45.9760118Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:45.9760744Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:45.9761170Z _lazy_init(state, module) 2022-11-23T03:54:45.9761744Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:45.9762188Z handle.init_flat_param_attributes() 2022-11-23T03:54:45.9762759Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:45.9763155Z return func(*args, **kwargs) 2022-11-23T03:54:45.9763764Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:45.9764182Z p_assert( 2022-11-23T03:54:45.9764713Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:45.9765122Z traceback.print_stack() 2022-11-23T03:54:45.9765542Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9765935Z File "", line 1, in 2022-11-23T03:54:45.9766333Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:45.9766735Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:45.9767140Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:45.9767543Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:45.9768132Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:45.9768507Z self.run() 2022-11-23T03:54:45.9768856Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:45.9769259Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:45.9769837Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:45.9770257Z self.run_test(test_name, pipe) 2022-11-23T03:54:45.9770832Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:45.9771223Z getattr(self, test_name)() 2022-11-23T03:54:45.9771732Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:45.9772091Z fn() 2022-11-23T03:54:45.9772591Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:45.9772979Z test(self, **param_kwargs) 2022-11-23T03:54:45.9773625Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:45.9774011Z return func(*args, **kwargs) 2022-11-23T03:54:45.9774402Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:45.9774751Z self.run_subtests( 2022-11-23T03:54:45.9775258Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:45.9775673Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:45.9776226Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:45.9776641Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:45.9777200Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:45.9777653Z output = model(*input) 2022-11-23T03:54:45.9778135Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:45.9778515Z return forward_call(*input, **kwargs) 2022-11-23T03:54:45.9779065Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:45.9779509Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:45.9780079Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:45.9780464Z _lazy_init(state, module) 2022-11-23T03:54:45.9780959Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:45.9781363Z handle.init_flat_param_attributes() 2022-11-23T03:54:45.9781882Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:45.9782269Z return func(*args, **kwargs) 2022-11-23T03:54:45.9782820Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:45.9783192Z p_assert( 2022-11-23T03:54:45.9783667Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:45.9784025Z traceback.print_stack() 2022-11-23T03:54:45.9784402Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9784776Z File "", line 1, in 2022-11-23T03:54:45.9785132Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:45.9785494Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:45.9785853Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:45.9786202Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:45.9786592Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:45.9786921Z self.run() 2022-11-23T03:54:45.9787250Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:45.9787610Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:45.9788128Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:45.9788510Z self.run_test(test_name, pipe) 2022-11-23T03:54:45.9789032Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:45.9789419Z getattr(self, test_name)() 2022-11-23T03:54:45.9789945Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:45.9790306Z fn() 2022-11-23T03:54:45.9790808Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:45.9791301Z test(self, **param_kwargs) 2022-11-23T03:54:45.9791814Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:45.9792204Z return func(*args, **kwargs) 2022-11-23T03:54:45.9792597Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:45.9792961Z self.run_subtests( 2022-11-23T03:54:45.9793469Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:45.9793890Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:45.9794445Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:45.9794842Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:45.9795463Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:45.9795863Z output = model(*input) 2022-11-23T03:54:45.9796354Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:45.9819349Z return forward_call(*input, **kwargs) 2022-11-23T03:54:45.9819939Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:45.9820388Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:45.9820946Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:45.9821333Z _lazy_init(state, module) 2022-11-23T03:54:45.9821849Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:45.9822243Z handle.init_flat_param_attributes() 2022-11-23T03:54:45.9822778Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:45.9823156Z return func(*args, **kwargs) 2022-11-23T03:54:45.9823706Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:45.9824062Z p_assert( 2022-11-23T03:54:45.9824539Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:45.9824911Z traceback.print_stack() 2022-11-23T03:54:45.9825290Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9825657Z File "", line 1, in 2022-11-23T03:54:45.9826019Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:45.9826367Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:45.9826732Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:45.9827098Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:45.9827480Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:45.9827807Z self.run() 2022-11-23T03:54:45.9828142Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:45.9828491Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:45.9829012Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:45.9829397Z self.run_test(test_name, pipe) 2022-11-23T03:54:45.9829928Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:45.9830317Z getattr(self, test_name)() 2022-11-23T03:54:45.9830845Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:45.9831325Z fn() 2022-11-23T03:54:45.9831823Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:45.9832213Z test(self, **param_kwargs) 2022-11-23T03:54:45.9832739Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:45.9833126Z return func(*args, **kwargs) 2022-11-23T03:54:45.9833519Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:45.9833882Z self.run_subtests( 2022-11-23T03:54:45.9834379Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:45.9834796Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:45.9835357Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:45.9835834Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:45.9836400Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:45.9836786Z output = model(*input) 2022-11-23T03:54:45.9837274Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:45.9837641Z return forward_call(*input, **kwargs) 2022-11-23T03:54:45.9838193Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:45.9838633Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:45.9839201Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:45.9839586Z _lazy_init(state, module) 2022-11-23T03:54:45.9840105Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:45.9840507Z handle.init_flat_param_attributes() 2022-11-23T03:54:45.9841014Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:45.9841387Z return func(*args, **kwargs) 2022-11-23T03:54:45.9841925Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:45.9842298Z p_assert( 2022-11-23T03:54:45.9842777Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:45.9843147Z traceback.print_stack() 2022-11-23T03:54:45.9843523Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9843876Z File "", line 1, in 2022-11-23T03:54:45.9844236Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:45.9844603Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:45.9844965Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:45.9845327Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:45.9845711Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:45.9846027Z self.run() 2022-11-23T03:54:45.9846354Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:45.9846717Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:45.9847235Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:45.9847615Z self.run_test(test_name, pipe) 2022-11-23T03:54:45.9848206Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:45.9848585Z getattr(self, test_name)() 2022-11-23T03:54:45.9849120Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:45.9849558Z fn() 2022-11-23T03:54:45.9850066Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:45.9850452Z test(self, **param_kwargs) 2022-11-23T03:54:45.9850973Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:45.9851354Z return func(*args, **kwargs) 2022-11-23T03:54:45.9851731Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:45.9852096Z self.run_subtests( 2022-11-23T03:54:45.9852606Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:45.9853021Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:45.9853630Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:45.9854048Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:45.9854619Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:45.9854989Z output = model(*input) 2022-11-23T03:54:45.9855478Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:45.9855863Z return forward_call(*input, **kwargs) 2022-11-23T03:54:45.9856416Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:45.9856861Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:45.9857431Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:45.9857816Z _lazy_init(state, module) 2022-11-23T03:54:45.9858323Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:45.9858722Z handle.init_flat_param_attributes() 2022-11-23T03:54:45.9859235Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:45.9859611Z return func(*args, **kwargs) 2022-11-23T03:54:45.9860156Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:45.9860529Z p_assert( 2022-11-23T03:54:45.9860996Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:45.9861366Z traceback.print_stack() 2022-11-23T03:54:45.9861745Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9862112Z File "", line 1, in 2022-11-23T03:54:45.9862475Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:45.9862842Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:45.9863193Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:45.9863557Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:45.9863941Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:45.9864269Z self.run() 2022-11-23T03:54:45.9864600Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:45.9864966Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:45.9865485Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:45.9865851Z self.run_test(test_name, pipe) 2022-11-23T03:54:45.9866387Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:45.9866854Z getattr(self, test_name)() 2022-11-23T03:54:45.9867372Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:45.9867731Z fn() 2022-11-23T03:54:45.9868237Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:45.9868609Z test(self, **param_kwargs) 2022-11-23T03:54:45.9869132Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:45.9869514Z return func(*args, **kwargs) 2022-11-23T03:54:45.9869905Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:45.9870270Z self.run_subtests( 2022-11-23T03:54:45.9870777Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:45.9871261Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:45.9871813Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:45.9872231Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:45.9872793Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:45.9873181Z output = model(*input) 2022-11-23T03:54:45.9873670Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:45.9874052Z return forward_call(*input, **kwargs) 2022-11-23T03:54:45.9874606Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:45.9875037Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:45.9875609Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:45.9875996Z _lazy_init(state, module) 2022-11-23T03:54:45.9876509Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:45.9876906Z handle.init_flat_param_attributes() 2022-11-23T03:54:45.9877425Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:45.9877804Z return func(*args, **kwargs) 2022-11-23T03:54:45.9878334Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:45.9878708Z p_assert( 2022-11-23T03:54:45.9879179Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:45.9879550Z traceback.print_stack() 2022-11-23T03:54:45.9879928Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9880301Z File "", line 1, in 2022-11-23T03:54:45.9880499Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:45.9880621Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:45.9880811Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:45.9880950Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:45.9881153Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:45.9881248Z self.run() 2022-11-23T03:54:45.9881441Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:45.9881577Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:45.9881925Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:45.9882050Z self.run_test(test_name, pipe) 2022-11-23T03:54:45.9882426Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:45.9882612Z getattr(self, test_name)() 2022-11-23T03:54:45.9882979Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:45.9883072Z fn() 2022-11-23T03:54:45.9883443Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:45.9883565Z test(self, **param_kwargs) 2022-11-23T03:54:45.9883929Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:45.9884046Z return func(*args, **kwargs) 2022-11-23T03:54:45.9884267Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:45.9884372Z self.run_subtests( 2022-11-23T03:54:45.9884913Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:45.9885073Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:45.9885450Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:45.9885595Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:45.9885980Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:45.9886092Z output = model(*input) 2022-11-23T03:54:45.9886424Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:45.9886559Z return forward_call(*input, **kwargs) 2022-11-23T03:54:45.9886943Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:45.9887111Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:45.9887497Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:45.9887612Z _lazy_init(state, module) 2022-11-23T03:54:45.9888111Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:45.9888245Z handle.init_flat_param_attributes() 2022-11-23T03:54:45.9888637Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:45.9888813Z return func(*args, **kwargs) 2022-11-23T03:54:45.9889290Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:45.9889425Z p_assert( 2022-11-23T03:54:45.9889951Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:45.9890095Z traceback.print_stack() 2022-11-23T03:54:45.9890369Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9890517Z File "", line 1, in 2022-11-23T03:54:45.9890751Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:45.9890906Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:45.9891133Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:45.9891298Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:45.9891537Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:45.9891647Z self.run() 2022-11-23T03:54:45.9891882Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:45.9892045Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:45.9892466Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:45.9892721Z self.run_test(test_name, pipe) 2022-11-23T03:54:45.9893176Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:45.9893305Z getattr(self, test_name)() 2022-11-23T03:54:45.9893740Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:45.9893853Z fn() 2022-11-23T03:54:45.9894295Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:45.9894436Z test(self, **param_kwargs) 2022-11-23T03:54:45.9894876Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:45.9895016Z return func(*args, **kwargs) 2022-11-23T03:54:45.9895300Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:45.9895496Z self.run_subtests( 2022-11-23T03:54:45.9895938Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:45.9896125Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:45.9896563Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:45.9896741Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:45.9897202Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:45.9897331Z output = model(*input) 2022-11-23T03:54:45.9897731Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:45.9897887Z return forward_call(*input, **kwargs) 2022-11-23T03:54:45.9898350Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:45.9898542Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:45.9898995Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:45.9899132Z _lazy_init(state, module) 2022-11-23T03:54:45.9899559Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:45.9899724Z handle.init_flat_param_attributes() 2022-11-23T03:54:45.9900131Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:45.9900272Z return func(*args, **kwargs) 2022-11-23T03:54:45.9900747Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:45.9900856Z p_assert( 2022-11-23T03:54:45.9901197Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:45.9901320Z traceback.print_stack() 2022-11-23T03:54:45.9901540Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9901661Z File "", line 1, in 2022-11-23T03:54:45.9901857Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:45.9901990Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:45.9902179Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:45.9902320Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:45.9902508Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:45.9902604Z self.run() 2022-11-23T03:54:45.9902796Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:45.9902933Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:45.9903340Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:45.9903462Z self.run_test(test_name, pipe) 2022-11-23T03:54:45.9903828Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:45.9903945Z getattr(self, test_name)() 2022-11-23T03:54:45.9904308Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:45.9904400Z fn() 2022-11-23T03:54:45.9904767Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:45.9904884Z test(self, **param_kwargs) 2022-11-23T03:54:45.9905247Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:45.9905369Z return func(*args, **kwargs) 2022-11-23T03:54:45.9905651Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:45.9905764Z self.run_subtests( 2022-11-23T03:54:45.9906122Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:45.9906262Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:45.9906631Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:45.9906774Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:45.9907153Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:45.9907265Z output = model(*input) 2022-11-23T03:54:45.9907597Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:45.9907731Z return forward_call(*input, **kwargs) 2022-11-23T03:54:45.9908120Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:45.9908284Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:45.9908657Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:45.9908771Z _lazy_init(state, module) 2022-11-23T03:54:45.9909126Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:45.9909260Z handle.init_flat_param_attributes() 2022-11-23T03:54:45.9909601Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:45.9909718Z return func(*args, **kwargs) 2022-11-23T03:54:45.9910099Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:45.9910201Z p_assert( 2022-11-23T03:54:45.9910540Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:45.9910655Z traceback.print_stack() 2022-11-23T03:54:45.9910861Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9910981Z File "", line 1, in 2022-11-23T03:54:45.9911177Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:45.9911309Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:45.9911499Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:45.9911641Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:45.9911841Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:45.9911936Z self.run() 2022-11-23T03:54:45.9912123Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:45.9912317Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:45.9912664Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:45.9912794Z self.run_test(test_name, pipe) 2022-11-23T03:54:45.9913165Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:45.9913283Z getattr(self, test_name)() 2022-11-23T03:54:45.9913647Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:45.9913738Z fn() 2022-11-23T03:54:45.9914092Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:45.9914208Z test(self, **param_kwargs) 2022-11-23T03:54:45.9914570Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:45.9914737Z return func(*args, **kwargs) 2022-11-23T03:54:45.9914972Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:45.9915077Z self.run_subtests( 2022-11-23T03:54:45.9915436Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:45.9915588Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:45.9915960Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:45.9916104Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:45.9916479Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:45.9916589Z output = model(*input) 2022-11-23T03:54:45.9916925Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:45.9917062Z return forward_call(*input, **kwargs) 2022-11-23T03:54:45.9917445Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:45.9917608Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:45.9917979Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:45.9918096Z _lazy_init(state, module) 2022-11-23T03:54:45.9918450Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:45.9918569Z handle.init_flat_param_attributes() 2022-11-23T03:54:45.9918911Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:45.9919027Z return func(*args, **kwargs) 2022-11-23T03:54:45.9919414Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:45.9919514Z p_assert( 2022-11-23T03:54:45.9919850Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:45.9919968Z traceback.print_stack() 2022-11-23T03:54:45.9920186Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:45.9920307Z File "", line 1, in 2022-11-23T03:54:45.9920501Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:45.9920635Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:45.9920826Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:45.9920965Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:45.9921167Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:45.9921319Z self.run() 2022-11-23T03:54:45.9921517Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:45.9921655Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:45.9921987Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:45.9922113Z self.run_test(test_name, pipe) 2022-11-23T03:54:45.9922484Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:45.9922603Z getattr(self, test_name)() 2022-11-23T03:54:45.9922965Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:45.9923057Z fn() 2022-11-23T03:54:45.9923425Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:45.9923542Z test(self, **param_kwargs) 2022-11-23T03:54:45.9923957Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:45.9924075Z return func(*args, **kwargs) 2022-11-23T03:54:45.9924307Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:45.9924414Z self.run_subtests( 2022-11-23T03:54:45.9924769Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:45.9924923Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:45.9925291Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0029519Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0030450Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0030632Z output = model(*input) 2022-11-23T03:54:46.0031100Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0031301Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0031919Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0032186Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0032846Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0033010Z _lazy_init(state, module) 2022-11-23T03:54:46.0033521Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0033672Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0034063Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0034203Z return func(*args, **kwargs) 2022-11-23T03:54:46.0034632Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0034741Z p_assert( 2022-11-23T03:54:46.0035116Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0035249Z traceback.print_stack() 2022-11-23T03:54:46.0035486Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0035623Z File "", line 1, in 2022-11-23T03:54:46.0035843Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0035998Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0036210Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0036371Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0036923Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0037033Z self.run() 2022-11-23T03:54:46.0037230Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0037383Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0037769Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0037913Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0038323Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0038453Z getattr(self, test_name)() 2022-11-23T03:54:46.0038861Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0038966Z fn() 2022-11-23T03:54:46.0039476Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0039617Z test(self, **param_kwargs) 2022-11-23T03:54:46.0040023Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0040154Z return func(*args, **kwargs) 2022-11-23T03:54:46.0040414Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:46.0040532Z self.run_subtests( 2022-11-23T03:54:46.0040929Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0041099Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0041506Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0041669Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0042076Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0042206Z output = model(*input) 2022-11-23T03:54:46.0042572Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0042722Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0043142Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0043325Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0043738Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0043866Z _lazy_init(state, module) 2022-11-23T03:54:46.0044261Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0044409Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0044795Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0044926Z return func(*args, **kwargs) 2022-11-23T03:54:46.0045350Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0045459Z p_assert( 2022-11-23T03:54:46.0045835Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0045967Z traceback.print_stack() 2022-11-23T03:54:46.0046206Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0046342Z File "", line 1, in 2022-11-23T03:54:46.0046545Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0046694Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0046908Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0047141Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0047365Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0047475Z self.run() 2022-11-23T03:54:46.0047868Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0048009Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0048368Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0048497Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0048863Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0048979Z getattr(self, test_name)() 2022-11-23T03:54:46.0049345Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0049434Z fn() 2022-11-23T03:54:46.0049886Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0050006Z test(self, **param_kwargs) 2022-11-23T03:54:46.0050358Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0050475Z return func(*args, **kwargs) 2022-11-23T03:54:46.0050706Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:46.0050811Z self.run_subtests( 2022-11-23T03:54:46.0051167Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0051321Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0051690Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0051837Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0052221Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0052329Z output = model(*input) 2022-11-23T03:54:46.0052660Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0052794Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0053174Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0053337Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0053707Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0053818Z _lazy_init(state, module) 2022-11-23T03:54:46.0054171Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0054307Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0054647Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0054750Z return func(*args, **kwargs) 2022-11-23T03:54:46.0055134Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0055226Z p_assert( 2022-11-23T03:54:46.0055562Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0055676Z traceback.print_stack() 2022-11-23T03:54:46.0055894Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0056016Z File "", line 1, in 2022-11-23T03:54:46.0056209Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0056339Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0056675Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0056815Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0057014Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0057105Z self.run() 2022-11-23T03:54:46.0057295Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0057428Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0057774Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0057897Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0058253Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0058367Z getattr(self, test_name)() 2022-11-23T03:54:46.0058774Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0058866Z fn() 2022-11-23T03:54:46.0059240Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0059353Z test(self, **param_kwargs) 2022-11-23T03:54:46.0059714Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0059832Z return func(*args, **kwargs) 2022-11-23T03:54:46.0060060Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:46.0060164Z self.run_subtests( 2022-11-23T03:54:46.0060516Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0060668Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0061034Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0061180Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0061562Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0061674Z output = model(*input) 2022-11-23T03:54:46.0062003Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0062137Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0062505Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0062672Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0063043Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0063155Z _lazy_init(state, module) 2022-11-23T03:54:46.0063518Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0063648Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0063988Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0064103Z return func(*args, **kwargs) 2022-11-23T03:54:46.0064480Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0064572Z p_assert( 2022-11-23T03:54:46.0064913Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0065026Z traceback.print_stack() 2022-11-23T03:54:46.0065240Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0065358Z File "", line 1, in 2022-11-23T03:54:46.0065555Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0065743Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0065931Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0066058Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0066257Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0066351Z self.run() 2022-11-23T03:54:46.0066538Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0066677Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0067020Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0067143Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0067509Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0067676Z getattr(self, test_name)() 2022-11-23T03:54:46.0068043Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0068130Z fn() 2022-11-23T03:54:46.0068497Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0068610Z test(self, **param_kwargs) 2022-11-23T03:54:46.0068967Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0069081Z return func(*args, **kwargs) 2022-11-23T03:54:46.0069310Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:46.0069413Z self.run_subtests( 2022-11-23T03:54:46.0069756Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0069911Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0070279Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0070421Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0070798Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0070907Z output = model(*input) 2022-11-23T03:54:46.0071234Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0071364Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0071743Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0071906Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0072277Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0072393Z _lazy_init(state, module) 2022-11-23T03:54:46.0072744Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0072876Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0073216Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0073332Z return func(*args, **kwargs) 2022-11-23T03:54:46.0073717Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0073812Z p_assert( 2022-11-23T03:54:46.0074150Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0074252Z traceback.print_stack() 2022-11-23T03:54:46.0074472Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0074655Z File "", line 1, in 2022-11-23T03:54:46.0074851Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0074983Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0075174Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0075316Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0075516Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0075611Z self.run() 2022-11-23T03:54:46.0075802Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0075937Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0076285Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0076410Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0076821Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0076942Z getattr(self, test_name)() 2022-11-23T03:54:46.0077306Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0077383Z fn() 2022-11-23T03:54:46.0077752Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0077868Z test(self, **param_kwargs) 2022-11-23T03:54:46.0078229Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0078344Z return func(*args, **kwargs) 2022-11-23T03:54:46.0078574Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:46.0078678Z self.run_subtests( 2022-11-23T03:54:46.0079037Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0079192Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0079559Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0079700Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0080081Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0080192Z output = model(*input) 2022-11-23T03:54:46.0080522Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0080653Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0081032Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0081198Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0081575Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0081688Z _lazy_init(state, module) 2022-11-23T03:54:46.0082030Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0082164Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0082507Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0082623Z return func(*args, **kwargs) 2022-11-23T03:54:46.0083006Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0083099Z p_assert( 2022-11-23T03:54:46.0083437Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0083552Z traceback.print_stack() 2022-11-23T03:54:46.0083831Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0083950Z File "", line 1, in 2022-11-23T03:54:46.0084144Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0084275Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0084462Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0084602Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0084805Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0084900Z self.run() 2022-11-23T03:54:46.0085089Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0085212Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0085558Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0085731Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0086103Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0086218Z getattr(self, test_name)() 2022-11-23T03:54:46.0086587Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0086677Z fn() 2022-11-23T03:54:46.0087043Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0087159Z test(self, **param_kwargs) 2022-11-23T03:54:46.0087520Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0087636Z return func(*args, **kwargs) 2022-11-23T03:54:46.0087958Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:46.0088072Z self.run_subtests( 2022-11-23T03:54:46.0088434Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0088588Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0088957Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0089100Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0089485Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0089582Z output = model(*input) 2022-11-23T03:54:46.0089913Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0090053Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0090434Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0090607Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0090979Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0091093Z _lazy_init(state, module) 2022-11-23T03:54:46.0091450Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0091582Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0091924Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0092041Z return func(*args, **kwargs) 2022-11-23T03:54:46.0092426Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0092520Z p_assert( 2022-11-23T03:54:46.0092866Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0093054Z traceback.print_stack() 2022-11-23T03:54:46.0093271Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0093391Z File "", line 1, in 2022-11-23T03:54:46.0093574Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0093708Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0093898Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0094036Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0094238Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0094335Z self.run() 2022-11-23T03:54:46.0094526Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0094661Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0095061Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0095190Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0095561Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0095678Z getattr(self, test_name)() 2022-11-23T03:54:46.0096042Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0096136Z fn() 2022-11-23T03:54:46.0096506Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0096622Z test(self, **param_kwargs) 2022-11-23T03:54:46.0096984Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0097087Z return func(*args, **kwargs) 2022-11-23T03:54:46.0097323Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:46.0097430Z self.run_subtests( 2022-11-23T03:54:46.0097787Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0097941Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0098311Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0098455Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0098838Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0098947Z output = model(*input) 2022-11-23T03:54:46.0099279Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0099411Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0099797Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0099962Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0100335Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0100449Z _lazy_init(state, module) 2022-11-23T03:54:46.0100805Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0100939Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0101284Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0101401Z return func(*args, **kwargs) 2022-11-23T03:54:46.0101772Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0101918Z p_assert( 2022-11-23T03:54:46.0102266Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0102381Z traceback.print_stack() 2022-11-23T03:54:46.0102600Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0102717Z File "", line 1, in 2022-11-23T03:54:46.0102913Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0103046Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0103235Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0103377Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0103579Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0103675Z self.run() 2022-11-23T03:54:46.0103868Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0104052Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0104407Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0104532Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0104899Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0105002Z getattr(self, test_name)() 2022-11-23T03:54:46.0105364Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0105456Z fn() 2022-11-23T03:54:46.0105824Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0105943Z test(self, **param_kwargs) 2022-11-23T03:54:46.0106307Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0106430Z return func(*args, **kwargs) 2022-11-23T03:54:46.0106662Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:46.0106767Z self.run_subtests( 2022-11-23T03:54:46.0107124Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0107278Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0107648Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0107789Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0108170Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0108279Z output = model(*input) 2022-11-23T03:54:46.0108611Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0108750Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0109131Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0109283Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0109655Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0109767Z _lazy_init(state, module) 2022-11-23T03:54:46.0110123Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0110255Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0110596Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0110712Z return func(*args, **kwargs) 2022-11-23T03:54:46.0111097Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0111242Z p_assert( 2022-11-23T03:54:46.0111580Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0111695Z traceback.print_stack() 2022-11-23T03:54:46.0111912Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0112126Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0112341Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0112555Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0112772Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0112982Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0113248Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0113453Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0113665Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0113880Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0114097Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0114307Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0114518Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0114732Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0114947Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0115160Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0115374Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0115583Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0115797Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0116006Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0116220Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0116431Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0116641Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0116855Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0117072Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0117280Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0117490Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0117701Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0117903Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0118112Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0118323Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0118536Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0118796Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0119008Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0119224Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0119432Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0119644Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0119853Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0120065Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0120275Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0120485Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0120743Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0120957Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0121162Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0121373Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0121582Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0121790Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0121893Z dist init r=0, world=2 2022-11-23T03:54:46.0122194Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.0122502Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.0122816Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.0123121Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.0123420Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.0123728Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.0124033Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.0124336Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.0124642Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.0124946Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.0125246Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.0125550Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.0125697Z dist init r=1, world=2 2022-11-23T03:54:46.0125998Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.0126297Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.0126598Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.0126901Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.0127197Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.0127538Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.0127955Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.0128258Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.0128559Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.0128855Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.0129157Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.0129463Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.0129552Z ok (63.734s) 2022-11-23T03:54:46.0129753Z test_delayed_optim_step_offload_true_shard_grad_op (__main__.TestParityWithDDP) 2022-11-23T03:54:46.0130047Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42939 2022-11-23T03:54:46.0130249Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42940 2022-11-23T03:54:46.0130639Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.0130802Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.0131187Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.0131366Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.0131587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.0131958Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.0132118Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.0132501Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.0132678Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.0132890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.0133294Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.0133772Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.0134046Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.0134320Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.0134533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.0134746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.0134962Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0135175Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0136286Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.0136399Z warnings.warn( 2022-11-23T03:54:46.0136519Z File "", line 1, in 2022-11-23T03:54:46.0136715Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0136852Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0137042Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0137180Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0137385Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0137478Z self.run() 2022-11-23T03:54:46.0137673Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0137811Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0138163Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0138276Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0138645Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0138759Z getattr(self, test_name)() 2022-11-23T03:54:46.0139126Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0139221Z fn() 2022-11-23T03:54:46.0139598Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0139714Z test(self, **param_kwargs) 2022-11-23T03:54:46.0140084Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0140205Z return func(*args, **kwargs) 2022-11-23T03:54:46.0140440Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:46.0140545Z self.run_subtests( 2022-11-23T03:54:46.0140905Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0141063Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0141434Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0141578Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0141962Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0142071Z output = model(*input) 2022-11-23T03:54:46.0142475Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0142595Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0142980Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0143148Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0143526Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0143640Z _lazy_init(state, module) 2022-11-23T03:54:46.0144001Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0144134Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0144480Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0144599Z return func(*args, **kwargs) 2022-11-23T03:54:46.0145045Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0145140Z p_assert( 2022-11-23T03:54:46.0145485Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0145600Z traceback.print_stack() 2022-11-23T03:54:46.0145821Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0146877Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.0146986Z warnings.warn( 2022-11-23T03:54:46.0147102Z File "", line 1, in 2022-11-23T03:54:46.0147300Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0147429Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0147616Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0147754Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0147943Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0148035Z self.run() 2022-11-23T03:54:46.0148226Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0148365Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0148713Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0148837Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0149210Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0149326Z getattr(self, test_name)() 2022-11-23T03:54:46.0149694Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0149784Z fn() 2022-11-23T03:54:46.0150160Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0150274Z test(self, **param_kwargs) 2022-11-23T03:54:46.0150640Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0150757Z return func(*args, **kwargs) 2022-11-23T03:54:46.0150989Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:46.0151094Z self.run_subtests( 2022-11-23T03:54:46.0151461Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0151657Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0152032Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0152174Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0152559Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0152669Z output = model(*input) 2022-11-23T03:54:46.0153004Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0153137Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0153529Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0153742Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0154129Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0154243Z _lazy_init(state, module) 2022-11-23T03:54:46.0154602Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0154737Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0155083Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0155205Z return func(*args, **kwargs) 2022-11-23T03:54:46.0155595Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0155688Z p_assert( 2022-11-23T03:54:46.0156029Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0156137Z traceback.print_stack() 2022-11-23T03:54:46.0156356Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0156476Z File "", line 1, in 2022-11-23T03:54:46.0156672Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0156806Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0156996Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0157139Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0157344Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0157439Z self.run() 2022-11-23T03:54:46.0157631Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0157775Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0158132Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0158258Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0158634Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0158749Z getattr(self, test_name)() 2022-11-23T03:54:46.0159113Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0159207Z fn() 2022-11-23T03:54:46.0159566Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0159683Z test(self, **param_kwargs) 2022-11-23T03:54:46.0160050Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0160169Z return func(*args, **kwargs) 2022-11-23T03:54:46.0160412Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:46.0160581Z self.run_subtests( 2022-11-23T03:54:46.0160948Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0161102Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0161479Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0161631Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0162017Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0162126Z output = model(*input) 2022-11-23T03:54:46.0162464Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0162596Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0163031Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0163207Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0163593Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0163716Z _lazy_init(state, module) 2022-11-23T03:54:46.0164081Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0164202Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0164547Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0164664Z return func(*args, **kwargs) 2022-11-23T03:54:46.0165057Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0165153Z p_assert( 2022-11-23T03:54:46.0165500Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0165620Z traceback.print_stack() 2022-11-23T03:54:46.0165837Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0165957Z File "", line 1, in 2022-11-23T03:54:46.0166160Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0166298Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0166490Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0166630Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0166833Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0166929Z self.run() 2022-11-23T03:54:46.0167130Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0167253Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0167612Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0167900Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0168280Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0168397Z getattr(self, test_name)() 2022-11-23T03:54:46.0168771Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0168864Z fn() 2022-11-23T03:54:46.0169241Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0169367Z test(self, **param_kwargs) 2022-11-23T03:54:46.0169737Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0169854Z return func(*args, **kwargs) 2022-11-23T03:54:46.0170167Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:46.0170282Z self.run_subtests( 2022-11-23T03:54:46.0170645Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0170797Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0171172Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0171315Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0171699Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0171796Z output = model(*input) 2022-11-23T03:54:46.0172139Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0172275Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0172714Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0172893Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0173277Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0173398Z _lazy_init(state, module) 2022-11-23T03:54:46.0173754Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0173892Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0174237Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0174359Z return func(*args, **kwargs) 2022-11-23T03:54:46.0174748Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0174849Z p_assert( 2022-11-23T03:54:46.0175187Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0175308Z traceback.print_stack() 2022-11-23T03:54:46.0175529Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0175656Z File "", line 1, in 2022-11-23T03:54:46.0175859Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0175980Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0176184Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0176331Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0176537Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0176640Z self.run() 2022-11-23T03:54:46.0176844Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0176986Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0177350Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0177482Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0177849Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0177965Z getattr(self, test_name)() 2022-11-23T03:54:46.0178333Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0178431Z fn() 2022-11-23T03:54:46.0178809Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0178937Z test(self, **param_kwargs) 2022-11-23T03:54:46.0179310Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0179490Z return func(*args, **kwargs) 2022-11-23T03:54:46.0179715Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:46.0179827Z self.run_subtests( 2022-11-23T03:54:46.0180190Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0180355Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0180735Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0180886Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0181266Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0181378Z output = model(*input) 2022-11-23T03:54:46.0181766Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0181903Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0182291Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0182467Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0182840Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0182955Z _lazy_init(state, module) 2022-11-23T03:54:46.0183326Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0183467Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0183809Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0183937Z return func(*args, **kwargs) 2022-11-23T03:54:46.0184336Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0184417Z p_assert( 2022-11-23T03:54:46.0184759Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0184881Z traceback.print_stack() 2022-11-23T03:54:46.0185107Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0185236Z File "", line 1, in 2022-11-23T03:54:46.0185433Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0185571Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0185770Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0185914Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0186121Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0186229Z self.run() 2022-11-23T03:54:46.0186419Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0186560Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0186913Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0187039Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0187407Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0187510Z getattr(self, test_name)() 2022-11-23T03:54:46.0187881Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0187979Z fn() 2022-11-23T03:54:46.0188362Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0188478Z test(self, **param_kwargs) 2022-11-23T03:54:46.0188929Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0189051Z return func(*args, **kwargs) 2022-11-23T03:54:46.0189294Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:46.0189403Z self.run_subtests( 2022-11-23T03:54:46.0189763Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0189919Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0190297Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0190449Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0190832Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0190988Z output = model(*input) 2022-11-23T03:54:46.0191329Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0191468Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0191850Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0192002Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0192384Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0192505Z _lazy_init(state, module) 2022-11-23T03:54:46.0192858Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0192995Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0193351Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0193476Z return func(*args, **kwargs) 2022-11-23T03:54:46.0193867Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0193964Z p_assert( 2022-11-23T03:54:46.0194308Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0194431Z traceback.print_stack() 2022-11-23T03:54:46.0194647Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0194770Z File "", line 1, in 2022-11-23T03:54:46.0194973Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0195110Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0195309Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0195458Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0195673Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0195756Z self.run() 2022-11-23T03:54:46.0195962Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0196102Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0196449Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0196581Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0196953Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0197072Z getattr(self, test_name)() 2022-11-23T03:54:46.0197444Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0197540Z fn() 2022-11-23T03:54:46.0197925Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0198101Z test(self, **param_kwargs) 2022-11-23T03:54:46.0198463Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0198581Z return func(*args, **kwargs) 2022-11-23T03:54:46.0198819Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:46.0198927Z self.run_subtests( 2022-11-23T03:54:46.0199285Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0199445Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0199802Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0199954Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0200392Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0200516Z output = model(*input) 2022-11-23T03:54:46.0200861Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0200995Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0201380Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0201558Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0201931Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0202046Z _lazy_init(state, module) 2022-11-23T03:54:46.0202406Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0202545Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0202899Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0203016Z return func(*args, **kwargs) 2022-11-23T03:54:46.0203409Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0203505Z p_assert( 2022-11-23T03:54:46.0203852Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0203966Z traceback.print_stack() 2022-11-23T03:54:46.0204173Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0204295Z File "", line 1, in 2022-11-23T03:54:46.0204498Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0204633Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0204827Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0204973Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0205187Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0205286Z self.run() 2022-11-23T03:54:46.0205482Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0205617Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0205959Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0206088Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0206463Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0206579Z getattr(self, test_name)() 2022-11-23T03:54:46.0206945Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0207099Z fn() 2022-11-23T03:54:46.0207480Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0207585Z test(self, **param_kwargs) 2022-11-23T03:54:46.0208021Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0208150Z return func(*args, **kwargs) 2022-11-23T03:54:46.0208388Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:46.0208497Z self.run_subtests( 2022-11-23T03:54:46.0208862Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0209016Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0209386Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0209595Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0209986Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0210104Z output = model(*input) 2022-11-23T03:54:46.0210439Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0210576Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0210962Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0211130Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0211505Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0211627Z _lazy_init(state, module) 2022-11-23T03:54:46.0211994Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0212140Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0212471Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0212590Z return func(*args, **kwargs) 2022-11-23T03:54:46.0212975Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0213078Z p_assert( 2022-11-23T03:54:46.0213417Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0213538Z traceback.print_stack() 2022-11-23T03:54:46.0213765Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0213888Z File "", line 1, in 2022-11-23T03:54:46.0214085Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0214227Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0214421Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0214565Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0214768Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0214870Z self.run() 2022-11-23T03:54:46.0215070Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0215206Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0215539Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0215663Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0216032Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0216154Z getattr(self, test_name)() 2022-11-23T03:54:46.0216532Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0216685Z fn() 2022-11-23T03:54:46.0217063Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0217186Z test(self, **param_kwargs) 2022-11-23T03:54:46.0217548Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0217669Z return func(*args, **kwargs) 2022-11-23T03:54:46.0217909Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:46.0218018Z self.run_subtests( 2022-11-23T03:54:46.0218383Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0218536Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0218949Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0219099Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0219493Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0219603Z output = model(*input) 2022-11-23T03:54:46.0219922Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0220063Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0220454Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0220620Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0220995Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0221121Z _lazy_init(state, module) 2022-11-23T03:54:46.0221484Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0221625Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0221971Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0222091Z return func(*args, **kwargs) 2022-11-23T03:54:46.0222483Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0222585Z p_assert( 2022-11-23T03:54:46.0222924Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0223044Z traceback.print_stack() 2022-11-23T03:54:46.0223266Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0223391Z File "", line 1, in 2022-11-23T03:54:46.0223602Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0223739Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0223915Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0224059Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0224269Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0224370Z self.run() 2022-11-23T03:54:46.0224561Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0224698Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0225053Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0225188Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0225567Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0225741Z getattr(self, test_name)() 2022-11-23T03:54:46.0226110Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0226209Z fn() 2022-11-23T03:54:46.0226582Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0226703Z test(self, **param_kwargs) 2022-11-23T03:54:46.0227066Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0227187Z return func(*args, **kwargs) 2022-11-23T03:54:46.0227436Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:46.0227527Z self.run_subtests( 2022-11-23T03:54:46.0227886Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0228113Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0228494Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0228644Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0229025Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0229139Z output = model(*input) 2022-11-23T03:54:46.0229480Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0229616Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0229996Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0230168Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0230554Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0230673Z _lazy_init(state, module) 2022-11-23T03:54:46.0231033Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0231170Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0231517Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0231636Z return func(*args, **kwargs) 2022-11-23T03:54:46.0232032Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0232113Z p_assert( 2022-11-23T03:54:46.0232455Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0232570Z traceback.print_stack() 2022-11-23T03:54:46.0232793Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0232922Z File "", line 1, in 2022-11-23T03:54:46.0233128Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0233267Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0233465Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0233615Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0233817Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0233913Z self.run() 2022-11-23T03:54:46.0234109Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0234252Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0234597Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0234723Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0235158Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0235281Z getattr(self, test_name)() 2022-11-23T03:54:46.0235634Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0235733Z fn() 2022-11-23T03:54:46.0236109Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0236227Z test(self, **param_kwargs) 2022-11-23T03:54:46.0236590Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0236711Z return func(*args, **kwargs) 2022-11-23T03:54:46.0236953Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:46.0237057Z self.run_subtests( 2022-11-23T03:54:46.0237463Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0237632Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0238006Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0238152Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0238544Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0238653Z output = model(*input) 2022-11-23T03:54:46.0238985Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0239122Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0239508Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0239675Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0240038Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0240153Z _lazy_init(state, module) 2022-11-23T03:54:46.0240522Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0240658Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0241004Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0241131Z return func(*args, **kwargs) 2022-11-23T03:54:46.0241514Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0241609Z p_assert( 2022-11-23T03:54:46.0241956Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0242070Z traceback.print_stack() 2022-11-23T03:54:46.0242303Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0242425Z File "", line 1, in 2022-11-23T03:54:46.0242624Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0242757Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0242947Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0243097Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0243301Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0243383Z self.run() 2022-11-23T03:54:46.0243586Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0243722Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0244066Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0244254Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0244637Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0244759Z getattr(self, test_name)() 2022-11-23T03:54:46.0245136Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0245227Z fn() 2022-11-23T03:54:46.0245603Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0245734Z test(self, **param_kwargs) 2022-11-23T03:54:46.0246096Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0246214Z return func(*args, **kwargs) 2022-11-23T03:54:46.0246448Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:46.0246607Z self.run_subtests( 2022-11-23T03:54:46.0246968Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0247127Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0247511Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0247641Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0248114Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0248231Z output = model(*input) 2022-11-23T03:54:46.0248571Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0248713Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0249100Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0249273Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0249655Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0249769Z _lazy_init(state, module) 2022-11-23T03:54:46.0250133Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0250272Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0250624Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0250749Z return func(*args, **kwargs) 2022-11-23T03:54:46.0251144Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0251237Z p_assert( 2022-11-23T03:54:46.0251585Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0251711Z traceback.print_stack() 2022-11-23T03:54:46.0251930Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0252037Z File "", line 1, in 2022-11-23T03:54:46.0252240Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0252379Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0252569Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0252714Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0252915Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0253011Z self.run() 2022-11-23T03:54:46.0253214Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0253355Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0253785Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0253914Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0254295Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0254412Z getattr(self, test_name)() 2022-11-23T03:54:46.0254786Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0254876Z fn() 2022-11-23T03:54:46.0255250Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0255377Z test(self, **param_kwargs) 2022-11-23T03:54:46.0255725Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0255847Z return func(*args, **kwargs) 2022-11-23T03:54:46.0256148Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:46.0256262Z self.run_subtests( 2022-11-23T03:54:46.0256634Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0256790Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0257171Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0257320Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0257713Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0257822Z output = model(*input) 2022-11-23T03:54:46.0258154Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0258290Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0258681Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0258859Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0259237Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0259356Z _lazy_init(state, module) 2022-11-23T03:54:46.0259719Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0259859Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0260189Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0260306Z return func(*args, **kwargs) 2022-11-23T03:54:46.0260704Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0260812Z p_assert( 2022-11-23T03:54:46.0261161Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0261276Z traceback.print_stack() 2022-11-23T03:54:46.0261497Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0261623Z File "", line 1, in 2022-11-23T03:54:46.0261827Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0261965Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0262158Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0262304Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0262518Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0262617Z self.run() 2022-11-23T03:54:46.0262812Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0263006Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0263361Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0263474Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0263846Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0263968Z getattr(self, test_name)() 2022-11-23T03:54:46.0264340Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0264441Z fn() 2022-11-23T03:54:46.0264815Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0264933Z test(self, **param_kwargs) 2022-11-23T03:54:46.0265298Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0265465Z return func(*args, **kwargs) 2022-11-23T03:54:46.0265711Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:46.0265813Z self.run_subtests( 2022-11-23T03:54:46.0266178Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0266335Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0266710Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0266859Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0267246Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0267367Z output = model(*input) 2022-11-23T03:54:46.0267713Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0267836Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0268219Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0268392Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0268768Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0268888Z _lazy_init(state, module) 2022-11-23T03:54:46.0269259Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0269393Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0269741Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0269862Z return func(*args, **kwargs) 2022-11-23T03:54:46.0270248Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0270345Z p_assert( 2022-11-23T03:54:46.0270695Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0270813Z traceback.print_stack() 2022-11-23T03:54:46.0271043Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0271169Z File "", line 1, in 2022-11-23T03:54:46.0271366Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0271501Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0271691Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0271820Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0272026Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0272188Z self.run() 2022-11-23T03:54:46.0272384Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0272519Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0272872Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0273002Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0273371Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0273493Z getattr(self, test_name)() 2022-11-23T03:54:46.0273862Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0273953Z fn() 2022-11-23T03:54:46.0274333Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0274456Z test(self, **param_kwargs) 2022-11-23T03:54:46.0274879Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0275010Z return func(*args, **kwargs) 2022-11-23T03:54:46.0275241Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:46.0275351Z self.run_subtests( 2022-11-23T03:54:46.0275699Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0275861Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0276233Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0276380Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0276771Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0276891Z output = model(*input) 2022-11-23T03:54:46.0277230Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0277362Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0277748Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0277922Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0278295Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0278412Z _lazy_init(state, module) 2022-11-23T03:54:46.0278777Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0278914Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0279259Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0279385Z return func(*args, **kwargs) 2022-11-23T03:54:46.0279770Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0279871Z p_assert( 2022-11-23T03:54:46.0280197Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0280327Z traceback.print_stack() 2022-11-23T03:54:46.0280550Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0280675Z File "", line 1, in 2022-11-23T03:54:46.0280877Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0281015Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0281213Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0281353Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0281755Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0281860Z self.run() 2022-11-23T03:54:46.0282053Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0282194Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0282546Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0282680Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0283048Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0283174Z getattr(self, test_name)() 2022-11-23T03:54:46.0283548Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0283626Z fn() 2022-11-23T03:54:46.0284046Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0284172Z test(self, **param_kwargs) 2022-11-23T03:54:46.0284546Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0284665Z return func(*args, **kwargs) 2022-11-23T03:54:46.0284903Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:46.0285012Z self.run_subtests( 2022-11-23T03:54:46.0285370Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0285529Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0285907Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0286054Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0286448Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0286573Z output = model(*input) 2022-11-23T03:54:46.0286912Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0287046Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0287433Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0287604Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0288372Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0288495Z _lazy_init(state, module) 2022-11-23T03:54:46.0288899Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0289041Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0289405Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0289531Z return func(*args, **kwargs) 2022-11-23T03:54:46.0289916Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0290012Z p_assert( 2022-11-23T03:54:46.0290364Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0290482Z traceback.print_stack() 2022-11-23T03:54:46.0290706Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0290827Z File "", line 1, in 2022-11-23T03:54:46.0291033Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0291173Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0291365Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0291595Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0291808Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0291904Z self.run() 2022-11-23T03:54:46.0292082Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0292223Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0292581Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0292709Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0293083Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0293202Z getattr(self, test_name)() 2022-11-23T03:54:46.0293573Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0293674Z fn() 2022-11-23T03:54:46.0294117Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0294238Z test(self, **param_kwargs) 2022-11-23T03:54:46.0294608Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0294732Z return func(*args, **kwargs) 2022-11-23T03:54:46.0294971Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:46.0295076Z self.run_subtests( 2022-11-23T03:54:46.0295440Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0295603Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0295977Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0296138Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0296504Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0296614Z output = model(*input) 2022-11-23T03:54:46.0296948Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0297087Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0297467Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0297639Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0298024Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0298138Z _lazy_init(state, module) 2022-11-23T03:54:46.0298500Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0298643Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0298985Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0299106Z return func(*args, **kwargs) 2022-11-23T03:54:46.0299502Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0299596Z p_assert( 2022-11-23T03:54:46.0299937Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0300062Z traceback.print_stack() 2022-11-23T03:54:46.0300283Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0300413Z File "", line 1, in 2022-11-23T03:54:46.0300597Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0300731Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0300985Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0301135Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0301339Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0301442Z self.run() 2022-11-23T03:54:46.0301641Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0301779Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0302132Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0302265Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0302636Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0302757Z getattr(self, test_name)() 2022-11-23T03:54:46.0303180Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0303281Z fn() 2022-11-23T03:54:46.0303665Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0303787Z test(self, **param_kwargs) 2022-11-23T03:54:46.0304162Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0304266Z return func(*args, **kwargs) 2022-11-23T03:54:46.0304498Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:46.0304608Z self.run_subtests( 2022-11-23T03:54:46.0304977Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0305131Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0305515Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0305670Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0306052Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0306163Z output = model(*input) 2022-11-23T03:54:46.0306504Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0306637Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0307025Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0307196Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0307574Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0307697Z _lazy_init(state, module) 2022-11-23T03:54:46.0308066Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0308209Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0308561Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0308665Z return func(*args, **kwargs) 2022-11-23T03:54:46.0309051Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0309148Z p_assert( 2022-11-23T03:54:46.0309491Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0309610Z traceback.print_stack() 2022-11-23T03:54:46.0309833Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0309961Z File "", line 1, in 2022-11-23T03:54:46.0310170Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0310360Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0310554Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0310704Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0310910Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0311015Z self.run() 2022-11-23T03:54:46.0311208Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0311345Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0311697Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0311831Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0312184Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0312352Z getattr(self, test_name)() 2022-11-23T03:54:46.0312728Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0312829Z fn() 2022-11-23T03:54:46.0313206Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0313328Z test(self, **param_kwargs) 2022-11-23T03:54:46.0313693Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0313809Z return func(*args, **kwargs) 2022-11-23T03:54:46.0314049Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:46.0314161Z self.run_subtests( 2022-11-23T03:54:46.0314529Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0314685Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0315065Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0315213Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0315599Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0315721Z output = model(*input) 2022-11-23T03:54:46.0316062Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0316195Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0316565Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0316731Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0317119Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0317237Z _lazy_init(state, module) 2022-11-23T03:54:46.0317603Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0317741Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0318094Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0318218Z return func(*args, **kwargs) 2022-11-23T03:54:46.0318618Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0318717Z p_assert( 2022-11-23T03:54:46.0319065Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0319182Z traceback.print_stack() 2022-11-23T03:54:46.0319406Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0319589Z File "", line 1, in 2022-11-23T03:54:46.0319788Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0319921Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0320123Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0320251Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0320454Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0320558Z self.run() 2022-11-23T03:54:46.0320764Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0320901Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0321256Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0321388Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0321814Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0321938Z getattr(self, test_name)() 2022-11-23T03:54:46.0322312Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0322404Z fn() 2022-11-23T03:54:46.0322792Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0322907Z test(self, **param_kwargs) 2022-11-23T03:54:46.0323274Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0323396Z return func(*args, **kwargs) 2022-11-23T03:54:46.0323632Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:46.0323746Z self.run_subtests( 2022-11-23T03:54:46.0324109Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0324252Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0324626Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0324777Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0325158Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0325271Z output = model(*input) 2022-11-23T03:54:46.0325613Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0325746Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0326127Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0326306Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0326685Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0326800Z _lazy_init(state, module) 2022-11-23T03:54:46.0327166Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0327314Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0327657Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0327829Z return func(*args, **kwargs) 2022-11-23T03:54:46.0328228Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0328321Z p_assert( 2022-11-23T03:54:46.0328677Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0328778Z traceback.print_stack() 2022-11-23T03:54:46.0329088Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0329218Z File "", line 1, in 2022-11-23T03:54:46.0329422Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0329557Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0329749Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0329898Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0330108Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0330214Z self.run() 2022-11-23T03:54:46.0330407Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0330543Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0330900Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0331089Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0331468Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0331592Z getattr(self, test_name)() 2022-11-23T03:54:46.0331968Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0332064Z fn() 2022-11-23T03:54:46.0332422Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0332550Z test(self, **param_kwargs) 2022-11-23T03:54:46.0332915Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0333037Z return func(*args, **kwargs) 2022-11-23T03:54:46.0333277Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:46.0333398Z self.run_subtests( 2022-11-23T03:54:46.0333760Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0333923Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0334296Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0334450Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0334832Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0334942Z output = model(*input) 2022-11-23T03:54:46.0335275Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0335413Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0335801Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0335970Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0336343Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0336467Z _lazy_init(state, module) 2022-11-23T03:54:46.0336812Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0336948Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0337298Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0337414Z return func(*args, **kwargs) 2022-11-23T03:54:46.0337802Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0337905Z p_assert( 2022-11-23T03:54:46.0338255Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0338435Z traceback.print_stack() 2022-11-23T03:54:46.0338664Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0338788Z File "", line 1, in 2022-11-23T03:54:46.0338993Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0339133Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0339328Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0339480Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0339683Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0339788Z self.run() 2022-11-23T03:54:46.0339983Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0340106Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0340508Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0340644Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0341023Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0341145Z getattr(self, test_name)() 2022-11-23T03:54:46.0341516Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0341610Z fn() 2022-11-23T03:54:46.0341981Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0342100Z test(self, **param_kwargs) 2022-11-23T03:54:46.0342466Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0342586Z return func(*args, **kwargs) 2022-11-23T03:54:46.0342829Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:46.0342941Z self.run_subtests( 2022-11-23T03:54:46.0343311Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0343471Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0343851Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0344001Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0344393Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0344490Z output = model(*input) 2022-11-23T03:54:46.0344826Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0344961Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0345345Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0345513Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0345896Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0346014Z _lazy_init(state, module) 2022-11-23T03:54:46.0346380Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0346517Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0346863Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0346982Z return func(*args, **kwargs) 2022-11-23T03:54:46.0347376Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0347538Z p_assert( 2022-11-23T03:54:46.0347896Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0348017Z traceback.print_stack() 2022-11-23T03:54:46.0348238Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0348372Z File "", line 1, in 2022-11-23T03:54:46.0348568Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0348688Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0348880Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0349028Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0349235Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0349340Z self.run() 2022-11-23T03:54:46.0349535Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0349727Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0350083Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0350219Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0350588Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0350713Z getattr(self, test_name)() 2022-11-23T03:54:46.0351078Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0351172Z fn() 2022-11-23T03:54:46.0351548Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0351664Z test(self, **param_kwargs) 2022-11-23T03:54:46.0352032Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0352166Z return func(*args, **kwargs) 2022-11-23T03:54:46.0352387Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:46.0352500Z self.run_subtests( 2022-11-23T03:54:46.0352865Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0353022Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0353401Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0353543Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0353934Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0354053Z output = model(*input) 2022-11-23T03:54:46.0354389Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0354524Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0354912Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0355090Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0355469Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0355587Z _lazy_init(state, module) 2022-11-23T03:54:46.0355960Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0356101Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0356444Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0356563Z return func(*args, **kwargs) 2022-11-23T03:54:46.0356937Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0357092Z p_assert( 2022-11-23T03:54:46.0357442Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0357562Z traceback.print_stack() 2022-11-23T03:54:46.0357781Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0357907Z File "", line 1, in 2022-11-23T03:54:46.0358115Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0358255Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0358462Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0358608Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0358820Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0358921Z self.run() 2022-11-23T03:54:46.0359168Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0359314Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0359669Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0359800Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0360170Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0360273Z getattr(self, test_name)() 2022-11-23T03:54:46.0360643Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0360746Z fn() 2022-11-23T03:54:46.0361121Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0361247Z test(self, **param_kwargs) 2022-11-23T03:54:46.0361617Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0361740Z return func(*args, **kwargs) 2022-11-23T03:54:46.0361981Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 193, in test_delayed_optim_step 2022-11-23T03:54:46.0362085Z self.run_subtests( 2022-11-23T03:54:46.0362451Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0362613Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0362982Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0363128Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0363518Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0363627Z output = model(*input) 2022-11-23T03:54:46.0363969Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0364109Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0364492Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0364643Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0365017Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0365131Z _lazy_init(state, module) 2022-11-23T03:54:46.0365492Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0365628Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0365978Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0366165Z return func(*args, **kwargs) 2022-11-23T03:54:46.0366562Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0366656Z p_assert( 2022-11-23T03:54:46.0367005Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0367130Z traceback.print_stack() 2022-11-23T03:54:46.0367356Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0367581Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0367914Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0368152Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0368367Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0368663Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0368892Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0369113Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0369315Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0369538Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0369750Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0369965Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0370186Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0370406Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0370622Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0370836Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0371059Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0371273Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0371487Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0371705Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0371924Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0372150Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0372371Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0372602Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0372818Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0373030Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0373246Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0373469Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0373668Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0373887Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0374112Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0374399Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0374621Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0374841Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0375063Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0375287Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0375499Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0375715Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0375935Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0376156Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0376419Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0376648Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0376864Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0377082Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0377296Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0377517Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0377738Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0377850Z dist init r=1, world=2 2022-11-23T03:54:46.0378154Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.0378471Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.0378790Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.0379099Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.0379414Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.0379724Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.0380035Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.0380344Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.0380668Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.0380973Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.0381283Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.0381600Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.0381749Z dist init r=0, world=2 2022-11-23T03:54:46.0382058Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.0382372Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.0382681Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.0382991Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.0383331Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.0383643Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.0383954Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.0384258Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.0384576Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.0384881Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.0385189Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.0385500Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.0385603Z ok (60.630s) 2022-11-23T03:54:46.0385819Z test_delayed_reduce_scatter_offload_false_no_shard (__main__.TestParityWithDDP) 2022-11-23T03:54:46.0386121Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43092 2022-11-23T03:54:46.0386332Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43093 2022-11-23T03:54:46.0386737Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.0386912Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.0387307Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.0387497Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.0387726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.0388111Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.0388278Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.0388668Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.0388856Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.0389073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.0389528Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.0389932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.0390218Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.0390496Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.0390723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.0390949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.0391172Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0391396Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0392534Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.0392639Z warnings.warn( 2022-11-23T03:54:46.0392856Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0393932Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.0394047Z warnings.warn( 2022-11-23T03:54:46.0394267Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0394488Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0394711Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0394933Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0395157Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0395371Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0395587Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0395813Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0396029Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0396248Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0396467Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0396677Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0396890Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0397092Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0397314Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0397525Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0397794Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0398003Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0398214Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0398999Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0399780Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0400596Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0401377Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0402157Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0402960Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0403193Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0403413Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0403616Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0403850Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0404075Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0404294Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0404530Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0404745Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0404960Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0405182Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0405402Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0405627Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0405845Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0406066Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0406292Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0406591Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0406819Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0407037Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0407250Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0407462Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0407680Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0408043Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0408244Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0408516Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0409316Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0410109Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0410874Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0411636Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0412401Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0413168Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0413952Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0414716Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0414943Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0415167Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0415271Z dist init r=1, world=2 2022-11-23T03:54:46.0415437Z dist init r=0, world=2 2022-11-23T03:54:46.0415538Z ok (8.340s) 2022-11-23T03:54:46.0415735Z test_delayed_reduce_scatter_offload_false_none (__main__.TestParityWithDDP) 2022-11-23T03:54:46.0416660Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82704 for platform(s) linux, rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.003s) 2022-11-23T03:54:46.0416884Z test_delayed_reduce_scatter_offload_false_shard_grad_op (__main__.TestParityWithDDP) 2022-11-23T03:54:46.0417828Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82398 for platform(s) linux, rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.002s) 2022-11-23T03:54:46.0418041Z test_delayed_reduce_scatter_offload_true_no_shard (__main__.TestParityWithDDP) 2022-11-23T03:54:46.0418349Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43245 2022-11-23T03:54:46.0418559Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43246 2022-11-23T03:54:46.0418928Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.0419101Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.0419486Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.0419669Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.0419910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.0420282Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.0420452Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.0420848Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.0421027Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.0421259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.0421670Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.0422066Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.0422352Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.0422637Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.0422854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.0423071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.0423300Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0423519Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0424588Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.0424750Z warnings.warn( 2022-11-23T03:54:46.0424878Z File "", line 1, in 2022-11-23T03:54:46.0425089Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0425224Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0425422Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0425550Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0425768Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0425865Z self.run() 2022-11-23T03:54:46.0426062Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0426208Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0426606Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0426739Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0427122Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0427241Z getattr(self, test_name)() 2022-11-23T03:54:46.0427623Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0427717Z fn() 2022-11-23T03:54:46.0428103Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0428225Z test(self, **param_kwargs) 2022-11-23T03:54:46.0428603Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0428729Z return func(*args, **kwargs) 2022-11-23T03:54:46.0428974Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:54:46.0429094Z self.run_subtests( 2022-11-23T03:54:46.0429440Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0429601Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0429981Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0430128Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0430519Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0430632Z output = model(*input) 2022-11-23T03:54:46.0430980Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0431124Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0431509Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0431688Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0432080Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0432199Z _lazy_init(state, module) 2022-11-23T03:54:46.0432566Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0432707Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0433068Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0433191Z return func(*args, **kwargs) 2022-11-23T03:54:46.0433588Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0433695Z p_assert( 2022-11-23T03:54:46.0434116Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0434218Z traceback.print_stack() 2022-11-23T03:54:46.0434453Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0435521Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.0435622Z warnings.warn( 2022-11-23T03:54:46.0435751Z File "", line 1, in 2022-11-23T03:54:46.0435964Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0436149Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0436351Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0436501Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0436705Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0436810Z self.run() 2022-11-23T03:54:46.0437005Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0437150Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0437513Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0437641Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0438025Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0438149Z getattr(self, test_name)() 2022-11-23T03:54:46.0438538Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0438639Z fn() 2022-11-23T03:54:46.0439000Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0439117Z test(self, **param_kwargs) 2022-11-23T03:54:46.0439494Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0439614Z return func(*args, **kwargs) 2022-11-23T03:54:46.0439859Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:54:46.0439973Z self.run_subtests( 2022-11-23T03:54:46.0440332Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0440490Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0440873Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0441016Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0441405Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0441526Z output = model(*input) 2022-11-23T03:54:46.0441864Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0442004Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0442390Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0442563Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0442946Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0443126Z _lazy_init(state, module) 2022-11-23T03:54:46.0443510Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0443668Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0444069Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0444199Z return func(*args, **kwargs) 2022-11-23T03:54:46.0444632Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0444762Z p_assert( 2022-11-23T03:54:46.0445137Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0445276Z traceback.print_stack() 2022-11-23T03:54:46.0445526Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0445665Z File "", line 1, in 2022-11-23T03:54:46.0445993Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0446163Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0446373Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0446541Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0446776Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0446892Z self.run() 2022-11-23T03:54:46.0447112Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0447248Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0447648Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0447921Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0448345Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0448474Z getattr(self, test_name)() 2022-11-23T03:54:46.0448842Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0448941Z fn() 2022-11-23T03:54:46.0449322Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0449439Z test(self, **param_kwargs) 2022-11-23T03:54:46.0449807Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0449932Z return func(*args, **kwargs) 2022-11-23T03:54:46.0450167Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:54:46.0450275Z self.run_subtests( 2022-11-23T03:54:46.0450643Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0450804Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0451174Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0451325Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0451703Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0451799Z output = model(*input) 2022-11-23T03:54:46.0452135Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0452277Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0452661Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0452835Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0453215Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0453421Z _lazy_init(state, module) 2022-11-23T03:54:46.0453790Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0453924Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0454276Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0454408Z return func(*args, **kwargs) 2022-11-23T03:54:46.0454794Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0454889Z p_assert( 2022-11-23T03:54:46.0455242Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0455358Z traceback.print_stack() 2022-11-23T03:54:46.0455629Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0455760Z File "", line 1, in 2022-11-23T03:54:46.0455958Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0456077Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0456278Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0456438Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0456645Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0456746Z self.run() 2022-11-23T03:54:46.0456951Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0457088Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0457449Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0457589Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0457963Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0458087Z getattr(self, test_name)() 2022-11-23T03:54:46.0458459Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0458558Z fn() 2022-11-23T03:54:46.0458943Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0459061Z test(self, **param_kwargs) 2022-11-23T03:54:46.0459434Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0459555Z return func(*args, **kwargs) 2022-11-23T03:54:46.0459779Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:54:46.0459893Z self.run_subtests( 2022-11-23T03:54:46.0460260Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0460426Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0460799Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0460953Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0461343Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0461461Z output = model(*input) 2022-11-23T03:54:46.0461801Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0461947Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0462340Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0462594Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0462981Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0463100Z _lazy_init(state, module) 2022-11-23T03:54:46.0463472Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0463619Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0463967Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0464093Z return func(*args, **kwargs) 2022-11-23T03:54:46.0464483Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0464563Z p_assert( 2022-11-23T03:54:46.0464912Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0465078Z traceback.print_stack() 2022-11-23T03:54:46.0465310Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0465433Z File "", line 1, in 2022-11-23T03:54:46.0465640Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0465783Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0465982Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0466130Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0466335Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0466447Z self.run() 2022-11-23T03:54:46.0466644Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0466786Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0467142Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0467287Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0467655Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0467758Z getattr(self, test_name)() 2022-11-23T03:54:46.0468127Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0468229Z fn() 2022-11-23T03:54:46.0468605Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0468722Z test(self, **param_kwargs) 2022-11-23T03:54:46.0469088Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0469214Z return func(*args, **kwargs) 2022-11-23T03:54:46.0469455Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:54:46.0469568Z self.run_subtests( 2022-11-23T03:54:46.0469927Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0470087Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0470468Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0470611Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0470994Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0471112Z output = model(*input) 2022-11-23T03:54:46.0471450Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0471592Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0471979Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0472202Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0472581Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0472702Z _lazy_init(state, module) 2022-11-23T03:54:46.0473060Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0473202Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0473545Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0473672Z return func(*args, **kwargs) 2022-11-23T03:54:46.0474068Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0474160Z p_assert( 2022-11-23T03:54:46.0474554Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0474683Z traceback.print_stack() 2022-11-23T03:54:46.0474902Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0475028Z File "", line 1, in 2022-11-23T03:54:46.0475237Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0475371Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0475566Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0475718Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0475924Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0476007Z self.run() 2022-11-23T03:54:46.0476210Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0476352Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0476707Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0476842Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0477211Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0477335Z getattr(self, test_name)() 2022-11-23T03:54:46.0477715Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0477812Z fn() 2022-11-23T03:54:46.0478194Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0478312Z test(self, **param_kwargs) 2022-11-23T03:54:46.0478684Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0478808Z return func(*args, **kwargs) 2022-11-23T03:54:46.0479056Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:54:46.0479169Z self.run_subtests( 2022-11-23T03:54:46.0479536Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0479701Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0480058Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0480204Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0480592Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0480715Z output = model(*input) 2022-11-23T03:54:46.0481058Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0481270Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0481658Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0481828Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0482205Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0482335Z _lazy_init(state, module) 2022-11-23T03:54:46.0482692Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0482829Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0483186Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0483306Z return func(*args, **kwargs) 2022-11-23T03:54:46.0483743Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0483846Z p_assert( 2022-11-23T03:54:46.0484189Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0484309Z traceback.print_stack() 2022-11-23T03:54:46.0484516Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0484652Z File "", line 1, in 2022-11-23T03:54:46.0484855Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0484995Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0485197Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0485340Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0485546Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0485651Z self.run() 2022-11-23T03:54:46.0485855Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0485994Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0486346Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0486475Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0486851Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0486969Z getattr(self, test_name)() 2022-11-23T03:54:46.0487336Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0487437Z fn() 2022-11-23T03:54:46.0487877Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0487982Z test(self, **param_kwargs) 2022-11-23T03:54:46.0488351Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0488476Z return func(*args, **kwargs) 2022-11-23T03:54:46.0488714Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:54:46.0488831Z self.run_subtests( 2022-11-23T03:54:46.0489200Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0489359Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0489737Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0489884Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0490281Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0490390Z output = model(*input) 2022-11-23T03:54:46.0490800Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0490939Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0491328Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0491497Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0491874Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0491993Z _lazy_init(state, module) 2022-11-23T03:54:46.0492358Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0492500Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0492830Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0492947Z return func(*args, **kwargs) 2022-11-23T03:54:46.0493387Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0493493Z p_assert( 2022-11-23T03:54:46.0493845Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0493962Z traceback.print_stack() 2022-11-23T03:54:46.0494184Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0494313Z File "", line 1, in 2022-11-23T03:54:46.0494510Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0494648Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0494848Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0494990Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0495205Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0495315Z self.run() 2022-11-23T03:54:46.0495509Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0495653Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0495987Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0496121Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0496491Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0496611Z getattr(self, test_name)() 2022-11-23T03:54:46.0496985Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0497082Z fn() 2022-11-23T03:54:46.0497461Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0497591Z test(self, **param_kwargs) 2022-11-23T03:54:46.0497955Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0498076Z return func(*args, **kwargs) 2022-11-23T03:54:46.0498322Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:54:46.0498427Z self.run_subtests( 2022-11-23T03:54:46.0498785Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0498945Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0499318Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0499471Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0499869Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0500042Z output = model(*input) 2022-11-23T03:54:46.0500360Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0500494Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0500889Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0501065Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0501447Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0501569Z _lazy_init(state, module) 2022-11-23T03:54:46.0501935Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0502076Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0502477Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0502603Z return func(*args, **kwargs) 2022-11-23T03:54:46.0502995Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0503090Z p_assert( 2022-11-23T03:54:46.0503429Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0503554Z traceback.print_stack() 2022-11-23T03:54:46.0503775Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0503902Z File "", line 1, in 2022-11-23T03:54:46.0504099Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0504235Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0504411Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0504573Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0504776Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0504878Z self.run() 2022-11-23T03:54:46.0505080Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0505218Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0505578Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0505704Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0506080Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0506210Z getattr(self, test_name)() 2022-11-23T03:54:46.0506576Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0506676Z fn() 2022-11-23T03:54:46.0507056Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0507180Z test(self, **param_kwargs) 2022-11-23T03:54:46.0507556Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0507673Z return func(*args, **kwargs) 2022-11-23T03:54:46.0507913Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:54:46.0508005Z self.run_subtests( 2022-11-23T03:54:46.0508372Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0508526Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0508902Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0509058Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0509516Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0509629Z output = model(*input) 2022-11-23T03:54:46.0509978Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0510116Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0510503Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0510680Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0511058Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0511180Z _lazy_init(state, module) 2022-11-23T03:54:46.0511549Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0511730Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0512082Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0512208Z return func(*args, **kwargs) 2022-11-23T03:54:46.0512598Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0512697Z p_assert( 2022-11-23T03:54:46.0513024Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0513157Z traceback.print_stack() 2022-11-23T03:54:46.0513378Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0513508Z File "", line 1, in 2022-11-23T03:54:46.0513714Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0513848Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0514052Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0514205Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0514412Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0514510Z self.run() 2022-11-23T03:54:46.0514708Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0514844Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0515195Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0515332Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0515700Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0515823Z getattr(self, test_name)() 2022-11-23T03:54:46.0516175Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0516281Z fn() 2022-11-23T03:54:46.0516654Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0516777Z test(self, **param_kwargs) 2022-11-23T03:54:46.0517151Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0517267Z return func(*args, **kwargs) 2022-11-23T03:54:46.0517507Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:54:46.0517624Z self.run_subtests( 2022-11-23T03:54:46.0517982Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0518138Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0518520Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0518731Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0519121Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0519243Z output = model(*input) 2022-11-23T03:54:46.0519577Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0519713Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0520103Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0520267Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0520645Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0520746Z _lazy_init(state, module) 2022-11-23T03:54:46.0521157Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0521298Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0521656Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0521786Z return func(*args, **kwargs) 2022-11-23T03:54:46.0522172Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0522271Z p_assert( 2022-11-23T03:54:46.0522618Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0522734Z traceback.print_stack() 2022-11-23T03:54:46.0522953Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0523075Z File "", line 1, in 2022-11-23T03:54:46.0523278Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0523425Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0523617Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0523766Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0523982Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0524079Z self.run() 2022-11-23T03:54:46.0524259Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0524402Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0524754Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0524881Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0525256Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0525385Z getattr(self, test_name)() 2022-11-23T03:54:46.0525762Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0525863Z fn() 2022-11-23T03:54:46.0526233Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0526354Z test(self, **param_kwargs) 2022-11-23T03:54:46.0526730Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0526851Z return func(*args, **kwargs) 2022-11-23T03:54:46.0527100Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:54:46.0527205Z self.run_subtests( 2022-11-23T03:54:46.0527571Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0527918Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0528618Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0528750Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0529139Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0529261Z output = model(*input) 2022-11-23T03:54:46.0529594Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0529729Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0530120Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0530295Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0530678Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0530872Z _lazy_init(state, module) 2022-11-23T03:54:46.0531238Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0531379Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0531722Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0531850Z return func(*args, **kwargs) 2022-11-23T03:54:46.0532240Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0532352Z p_assert( 2022-11-23T03:54:46.0532691Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0532815Z traceback.print_stack() 2022-11-23T03:54:46.0533046Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0533154Z File "", line 1, in 2022-11-23T03:54:46.0533355Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0533493Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0533693Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0533839Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0534052Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0534154Z self.run() 2022-11-23T03:54:46.0534354Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0534496Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0534846Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0534982Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0535354Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0535479Z getattr(self, test_name)() 2022-11-23T03:54:46.0535852Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0535952Z fn() 2022-11-23T03:54:46.0536332Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0536448Z test(self, **param_kwargs) 2022-11-23T03:54:46.0536799Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0536916Z return func(*args, **kwargs) 2022-11-23T03:54:46.0537156Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:54:46.0537273Z self.run_subtests( 2022-11-23T03:54:46.0537635Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0537870Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0538244Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0538398Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0538790Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0538904Z output = model(*input) 2022-11-23T03:54:46.0539247Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0539385Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0539772Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0539952Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0540382Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0540506Z _lazy_init(state, module) 2022-11-23T03:54:46.0540872Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0541014Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0541364Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0541469Z return func(*args, **kwargs) 2022-11-23T03:54:46.0541853Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0541951Z p_assert( 2022-11-23T03:54:46.0542301Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0542422Z traceback.print_stack() 2022-11-23T03:54:46.0542657Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0542789Z File "", line 1, in 2022-11-23T03:54:46.0542991Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0543137Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0543327Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0543475Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0543685Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0543781Z self.run() 2022-11-23T03:54:46.0543974Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0544115Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0544468Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0544581Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0544953Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0545077Z getattr(self, test_name)() 2022-11-23T03:54:46.0545454Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0545545Z fn() 2022-11-23T03:54:46.0545920Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0546045Z test(self, **param_kwargs) 2022-11-23T03:54:46.0546410Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0546530Z return func(*args, **kwargs) 2022-11-23T03:54:46.0546774Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:54:46.0546878Z self.run_subtests( 2022-11-23T03:54:46.0547310Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0547469Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0547837Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0547986Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0548376Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0548488Z output = model(*input) 2022-11-23T03:54:46.0548826Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0548945Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0549336Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0549554Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0549941Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0550066Z _lazy_init(state, module) 2022-11-23T03:54:46.0550422Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0550562Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0550920Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0551038Z return func(*args, **kwargs) 2022-11-23T03:54:46.0551425Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0551525Z p_assert( 2022-11-23T03:54:46.0551877Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0552006Z traceback.print_stack() 2022-11-23T03:54:46.0552238Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0552361Z File "", line 1, in 2022-11-23T03:54:46.0552563Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0552702Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0552895Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0553024Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0553234Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0553333Z self.run() 2022-11-23T03:54:46.0553531Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0553679Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0554028Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0554161Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0554540Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0554658Z getattr(self, test_name)() 2022-11-23T03:54:46.0555024Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0555123Z fn() 2022-11-23T03:54:46.0555503Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0555630Z test(self, **param_kwargs) 2022-11-23T03:54:46.0555993Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0556111Z return func(*args, **kwargs) 2022-11-23T03:54:46.0556355Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:54:46.0556542Z self.run_subtests( 2022-11-23T03:54:46.0556889Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0557046Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0557417Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0557571Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0557957Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0558077Z output = model(*input) 2022-11-23T03:54:46.0558412Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0558553Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0558985Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0559157Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0559543Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0559669Z _lazy_init(state, module) 2022-11-23T03:54:46.0560030Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0560172Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0560528Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0560646Z return func(*args, **kwargs) 2022-11-23T03:54:46.0561033Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0561137Z p_assert( 2022-11-23T03:54:46.0561487Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0561591Z traceback.print_stack() 2022-11-23T03:54:46.0561818Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0561951Z File "", line 1, in 2022-11-23T03:54:46.0562149Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0562290Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0562490Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0562641Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0562855Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0562952Z self.run() 2022-11-23T03:54:46.0563149Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0563300Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0563645Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0563780Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0564165Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0564282Z getattr(self, test_name)() 2022-11-23T03:54:46.0564651Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0564730Z fn() 2022-11-23T03:54:46.0565114Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0565232Z test(self, **param_kwargs) 2022-11-23T03:54:46.0565603Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0565800Z return func(*args, **kwargs) 2022-11-23T03:54:46.0566038Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:54:46.0566145Z self.run_subtests( 2022-11-23T03:54:46.0566509Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0566670Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0567040Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0567185Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0567572Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0567685Z output = model(*input) 2022-11-23T03:54:46.0568101Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0568454Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0568844Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0569016Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0569396Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0569509Z _lazy_init(state, module) 2022-11-23T03:54:46.0569851Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0569991Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0570342Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0570462Z return func(*args, **kwargs) 2022-11-23T03:54:46.0570854Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0570960Z p_assert( 2022-11-23T03:54:46.0571298Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0571419Z traceback.print_stack() 2022-11-23T03:54:46.0571647Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0571767Z File "", line 1, in 2022-11-23T03:54:46.0571971Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0572114Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0572309Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0572460Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0572666Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0572770Z self.run() 2022-11-23T03:54:46.0572955Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0573101Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0573446Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0573579Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0573956Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0574074Z getattr(self, test_name)() 2022-11-23T03:54:46.0574445Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0574547Z fn() 2022-11-23T03:54:46.0574916Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0575038Z test(self, **param_kwargs) 2022-11-23T03:54:46.0575414Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0575601Z return func(*args, **kwargs) 2022-11-23T03:54:46.0575841Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:54:46.0575947Z self.run_subtests( 2022-11-23T03:54:46.0576310Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0576477Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0576852Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0577000Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0577367Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0577494Z output = model(*input) 2022-11-23T03:54:46.0577884Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0578031Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0578411Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0578587Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0578966Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0579091Z _lazy_init(state, module) 2022-11-23T03:54:46.0579454Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0579594Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0579941Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0580076Z return func(*args, **kwargs) 2022-11-23T03:54:46.0580466Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0580576Z p_assert( 2022-11-23T03:54:46.0580920Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0581042Z traceback.print_stack() 2022-11-23T03:54:46.0581263Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0581390Z File "", line 1, in 2022-11-23T03:54:46.0581572Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0581711Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0581913Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0582060Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0582269Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0582368Z self.run() 2022-11-23T03:54:46.0582562Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0582705Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0583057Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0583186Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0583565Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0583687Z getattr(self, test_name)() 2022-11-23T03:54:46.0584056Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0584158Z fn() 2022-11-23T03:54:46.0584526Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0584717Z test(self, **param_kwargs) 2022-11-23T03:54:46.0585095Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0585199Z return func(*args, **kwargs) 2022-11-23T03:54:46.0585438Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:54:46.0585553Z self.run_subtests( 2022-11-23T03:54:46.0585910Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0586069Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0586447Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0586592Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0587041Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0587165Z output = model(*input) 2022-11-23T03:54:46.0587500Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0587637Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0588028Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0588194Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0588571Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0588697Z _lazy_init(state, module) 2022-11-23T03:54:46.0589055Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0589191Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0589548Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0589679Z return func(*args, **kwargs) 2022-11-23T03:54:46.0590050Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0590160Z p_assert( 2022-11-23T03:54:46.0590501Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0590623Z traceback.print_stack() 2022-11-23T03:54:46.0590851Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0590977Z File "", line 1, in 2022-11-23T03:54:46.0591188Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0591324Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0591525Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0591680Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0591882Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0591980Z self.run() 2022-11-23T03:54:46.0592178Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0592315Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0592667Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0592806Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0593160Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0593279Z getattr(self, test_name)() 2022-11-23T03:54:46.0593648Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0593751Z fn() 2022-11-23T03:54:46.0594192Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0594313Z test(self, **param_kwargs) 2022-11-23T03:54:46.0594683Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0594808Z return func(*args, **kwargs) 2022-11-23T03:54:46.0595060Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:54:46.0595184Z self.run_subtests( 2022-11-23T03:54:46.0595546Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0595711Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0596090Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0596236Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0596664Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0596779Z output = model(*input) 2022-11-23T03:54:46.0597124Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0597271Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0597654Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0597806Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0598182Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0598307Z _lazy_init(state, module) 2022-11-23T03:54:46.0598668Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0598813Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0599159Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0599291Z return func(*args, **kwargs) 2022-11-23T03:54:46.0599674Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0599772Z p_assert( 2022-11-23T03:54:46.0600121Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0600239Z traceback.print_stack() 2022-11-23T03:54:46.0600460Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0600584Z File "", line 1, in 2022-11-23T03:54:46.0600785Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0600923Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0601132Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0601276Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0601463Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0601568Z self.run() 2022-11-23T03:54:46.0601761Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0601899Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0602248Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0602383Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0602759Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0602877Z getattr(self, test_name)() 2022-11-23T03:54:46.0603247Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0603414Z fn() 2022-11-23T03:54:46.0603796Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0603919Z test(self, **param_kwargs) 2022-11-23T03:54:46.0604286Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0604404Z return func(*args, **kwargs) 2022-11-23T03:54:46.0604649Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:54:46.0604762Z self.run_subtests( 2022-11-23T03:54:46.0605120Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0605261Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0605677Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0605839Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0606225Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0606343Z output = model(*input) 2022-11-23T03:54:46.0606684Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0606819Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0607203Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0607384Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0607946Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0608163Z _lazy_init(state, module) 2022-11-23T03:54:46.0608610Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0608756Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0609106Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0609231Z return func(*args, **kwargs) 2022-11-23T03:54:46.0609618Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0609711Z p_assert( 2022-11-23T03:54:46.0610053Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0610178Z traceback.print_stack() 2022-11-23T03:54:46.0610383Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0610504Z File "", line 1, in 2022-11-23T03:54:46.0610720Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0610854Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0611058Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0611211Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0611413Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0611509Z self.run() 2022-11-23T03:54:46.0611705Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0611849Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0612201Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0612334Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0612704Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0612825Z getattr(self, test_name)() 2022-11-23T03:54:46.0613289Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0613381Z fn() 2022-11-23T03:54:46.0613737Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0613858Z test(self, **param_kwargs) 2022-11-23T03:54:46.0614233Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0614349Z return func(*args, **kwargs) 2022-11-23T03:54:46.0614593Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:54:46.0614710Z self.run_subtests( 2022-11-23T03:54:46.0615070Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0615229Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0615667Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0615816Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0616210Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0616329Z output = model(*input) 2022-11-23T03:54:46.0616668Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0616811Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0617195Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0617361Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0617745Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0617872Z _lazy_init(state, module) 2022-11-23T03:54:46.0618238Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0618359Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0618713Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0618832Z return func(*args, **kwargs) 2022-11-23T03:54:46.0619228Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0619334Z p_assert( 2022-11-23T03:54:46.0619676Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0619799Z traceback.print_stack() 2022-11-23T03:54:46.0620029Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0620155Z File "", line 1, in 2022-11-23T03:54:46.0620352Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0620494Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0620691Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0620835Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0621046Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0621143Z self.run() 2022-11-23T03:54:46.0621338Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0621460Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0621813Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0621938Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0622310Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0622502Z getattr(self, test_name)() 2022-11-23T03:54:46.0622870Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0622963Z fn() 2022-11-23T03:54:46.0623344Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0623465Z test(self, **param_kwargs) 2022-11-23T03:54:46.0623834Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0623961Z return func(*args, **kwargs) 2022-11-23T03:54:46.0624199Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:54:46.0624310Z self.run_subtests( 2022-11-23T03:54:46.0624712Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0624873Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0625253Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0625400Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0625794Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0625916Z output = model(*input) 2022-11-23T03:54:46.0626235Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0626378Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0626763Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0626936Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0627327Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0627451Z _lazy_init(state, module) 2022-11-23T03:54:46.0627810Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0627944Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0628297Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0628420Z return func(*args, **kwargs) 2022-11-23T03:54:46.0628806Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0628906Z p_assert( 2022-11-23T03:54:46.0629260Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0629379Z traceback.print_stack() 2022-11-23T03:54:46.0629602Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0629725Z File "", line 1, in 2022-11-23T03:54:46.0629932Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0630052Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0630244Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0630385Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0630590Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0630693Z self.run() 2022-11-23T03:54:46.0630888Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0631029Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0631386Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0631576Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0631950Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0632075Z getattr(self, test_name)() 2022-11-23T03:54:46.0632440Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0632532Z fn() 2022-11-23T03:54:46.0632912Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0633044Z test(self, **param_kwargs) 2022-11-23T03:54:46.0633407Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0633526Z return func(*args, **kwargs) 2022-11-23T03:54:46.0633750Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:54:46.0633870Z self.run_subtests( 2022-11-23T03:54:46.0634271Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0634427Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0634808Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0634958Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0635344Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0635463Z output = model(*input) 2022-11-23T03:54:46.0635798Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0635935Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0636328Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0636505Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0636884Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0637007Z _lazy_init(state, module) 2022-11-23T03:54:46.0637364Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0637504Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0637860Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0637978Z return func(*args, **kwargs) 2022-11-23T03:54:46.0638373Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0638453Z p_assert( 2022-11-23T03:54:46.0638810Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0638930Z traceback.print_stack() 2022-11-23T03:54:46.0639153Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0639283Z File "", line 1, in 2022-11-23T03:54:46.0639481Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0639614Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0639812Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0639959Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0640165Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0640276Z self.run() 2022-11-23T03:54:46.0640467Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0640603Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0640961Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0641162Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0641538Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0641641Z getattr(self, test_name)() 2022-11-23T03:54:46.0642020Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0642116Z fn() 2022-11-23T03:54:46.0642493Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0642624Z test(self, **param_kwargs) 2022-11-23T03:54:46.0642989Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0643108Z return func(*args, **kwargs) 2022-11-23T03:54:46.0643403Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:54:46.0643515Z self.run_subtests( 2022-11-23T03:54:46.0643881Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0644041Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0644419Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0644566Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0644958Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0645069Z output = model(*input) 2022-11-23T03:54:46.0645407Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0645555Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0645946Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0646117Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0646478Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0646596Z _lazy_init(state, module) 2022-11-23T03:54:46.0646953Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0647091Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0647447Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0647567Z return func(*args, **kwargs) 2022-11-23T03:54:46.0648039Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0648143Z p_assert( 2022-11-23T03:54:46.0648496Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0648612Z traceback.print_stack() 2022-11-23T03:54:46.0648835Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0648965Z File "", line 1, in 2022-11-23T03:54:46.0649171Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0649320Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0649510Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0649655Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0649871Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0649957Z self.run() 2022-11-23T03:54:46.0650150Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0650365Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0650722Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0650849Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0651222Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0651349Z getattr(self, test_name)() 2022-11-23T03:54:46.0651718Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0651822Z fn() 2022-11-23T03:54:46.0652193Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0652308Z test(self, **param_kwargs) 2022-11-23T03:54:46.0652677Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0652861Z return func(*args, **kwargs) 2022-11-23T03:54:46.0653107Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 214, in test_delayed_reduce_scatter 2022-11-23T03:54:46.0653224Z self.run_subtests( 2022-11-23T03:54:46.0653588Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0653750Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0654109Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0654262Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0654645Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0654763Z output = model(*input) 2022-11-23T03:54:46.0655113Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0655251Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0655636Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0655809Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0656182Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0656300Z _lazy_init(state, module) 2022-11-23T03:54:46.0656666Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0656799Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0657147Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0657274Z return func(*args, **kwargs) 2022-11-23T03:54:46.0657665Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0657761Z p_assert( 2022-11-23T03:54:46.0658105Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0658228Z traceback.print_stack() 2022-11-23T03:54:46.0658449Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0658651Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0658879Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0659105Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0659320Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0659536Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0659815Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0660035Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0660249Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0660462Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0660682Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0660903Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0661121Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0661342Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0661604Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0661826Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0662050Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0662273Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0662488Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0662707Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0662931Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0663130Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0663346Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0663566Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0663793Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0664008Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0664223Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0664443Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0664662Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0664888Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0665114Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0665328Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0665549Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0665762Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0665986Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0666198Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0666419Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0666644Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0666863Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0667658Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0668486Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0668703Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0668922Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0669152Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0669370Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0669590Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0669848Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0670079Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0670292Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.0670396Z dist init r=1, world=2 2022-11-23T03:54:46.0670719Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.0671043Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.0671352Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.0671669Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.0671984Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.0672291Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.0672596Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.0672913Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.0673221Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.0673527Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.0673850Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.0674160Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.0674271Z dist init r=0, world=2 2022-11-23T03:54:46.0674576Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.0674883Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.0675229Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.0675535Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.0675843Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.0676149Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.0676463Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.0676807Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.0677112Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.0677414Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.0677725Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.0678027Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.0678134Z ok (8.535s) 2022-11-23T03:54:46.0678337Z test_delayed_reduce_scatter_offload_true_none (__main__.TestParityWithDDP) 2022-11-23T03:54:46.0679264Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82399 for platform(s) linux, rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.002s) 2022-11-23T03:54:46.0679484Z test_delayed_reduce_scatter_offload_true_shard_grad_op (__main__.TestParityWithDDP) 2022-11-23T03:54:46.0680389Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82403 for platform(s) linux, rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s) 2022-11-23T03:54:46.0680731Z test_mixture_of_experts_offload_false_no_shard_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43398 2022-11-23T03:54:46.0680954Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43399 2022-11-23T03:54:46.0681331Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.0681501Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.0681901Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.0682082Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.0682310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.0682696Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.0682916Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.0683308Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.0683473Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.0683714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.0684119Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.0684517Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.0684803Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.0685091Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.0685364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.0685599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.0686665Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.0686769Z warnings.warn( 2022-11-23T03:54:46.0687005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T03:54:46.0688356Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.0688468Z warnings.warn( 2022-11-23T03:54:46.0688702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T03:54:46.0689114Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:54:46.0689519Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:54:46.0689756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T03:54:46.0690155Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T03:54:46.0690384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T03:54:46.0690792Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T03:54:46.0691012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T03:54:46.0691404Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T03:54:46.0691630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T03:54:46.0692035Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T03:54:46.0692816Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0693664Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0693895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1 2022-11-23T03:54:46.0694118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0 2022-11-23T03:54:46.0694517Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 2 nodes. 2022-11-23T03:54:46.0694969Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 2 nodes. 2022-11-23T03:54:46.0695197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1 2022-11-23T03:54:46.0695420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0 2022-11-23T03:54:46.0695825Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 2 nodes. 2022-11-23T03:54:46.0696232Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 2 nodes. 2022-11-23T03:54:46.0696453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0 2022-11-23T03:54:46.0696674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1 2022-11-23T03:54:46.0697078Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 2 nodes. 2022-11-23T03:54:46.0697458Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 2 nodes. 2022-11-23T03:54:46.0698248Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0699018Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0699790Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0700563Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0701336Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0702105Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0702395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 0 2022-11-23T03:54:46.0702621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 1 2022-11-23T03:54:46.0703017Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:8 with 2 nodes. 2022-11-23T03:54:46.0703415Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:8 with 2 nodes. 2022-11-23T03:54:46.0703649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 0 2022-11-23T03:54:46.0704088Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:9 with 2 nodes. 2022-11-23T03:54:46.0704320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 1 2022-11-23T03:54:46.0704712Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:9 with 2 nodes. 2022-11-23T03:54:46.0704938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 1 2022-11-23T03:54:46.0705147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 0 2022-11-23T03:54:46.0705548Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:10 with 2 nodes. 2022-11-23T03:54:46.0705951Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:10 with 2 nodes. 2022-11-23T03:54:46.0706719Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0706955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 0 2022-11-23T03:54:46.0707726Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0707953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 1 2022-11-23T03:54:46.0708354Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:11 with 2 nodes. 2022-11-23T03:54:46.0708761Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:11 with 2 nodes. 2022-11-23T03:54:46.0708991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 0 2022-11-23T03:54:46.0709220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 1 2022-11-23T03:54:46.0709617Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:12 with 2 nodes. 2022-11-23T03:54:46.0710020Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:12 with 2 nodes. 2022-11-23T03:54:46.0710244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 0 2022-11-23T03:54:46.0710645Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:13 with 2 nodes. 2022-11-23T03:54:46.0710933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 1 2022-11-23T03:54:46.0711327Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:13 with 2 nodes. 2022-11-23T03:54:46.0712093Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0712855Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0713664Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0713901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 1 2022-11-23T03:54:46.0714135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 0 2022-11-23T03:54:46.0714533Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:14 with 2 nodes. 2022-11-23T03:54:46.0714928Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:14 with 2 nodes. 2022-11-23T03:54:46.0715713Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0715939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 0 2022-11-23T03:54:46.0716161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 1 2022-11-23T03:54:46.0716565Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:15 with 2 nodes. 2022-11-23T03:54:46.0716966Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:15 with 2 nodes. 2022-11-23T03:54:46.0717193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 1 2022-11-23T03:54:46.0717428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 0 2022-11-23T03:54:46.0717828Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:16 with 2 nodes. 2022-11-23T03:54:46.0718234Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:16 with 2 nodes. 2022-11-23T03:54:46.0719007Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0719763Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0720535Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0721353Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0722111Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0722911Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0723152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 0 2022-11-23T03:54:46.0723374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 1 2022-11-23T03:54:46.0723780Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:17 with 2 nodes. 2022-11-23T03:54:46.0724185Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:17 with 2 nodes. 2022-11-23T03:54:46.0724952Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0725714Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0726485Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0726713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 1 2022-11-23T03:54:46.0726946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 0 2022-11-23T03:54:46.0727352Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:18 with 2 nodes. 2022-11-23T03:54:46.0727876Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:18 with 2 nodes. 2022-11-23T03:54:46.0728642Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0728876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 0 2022-11-23T03:54:46.0729110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 1 2022-11-23T03:54:46.0729520Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:19 with 2 nodes. 2022-11-23T03:54:46.0729978Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:19 with 2 nodes. 2022-11-23T03:54:46.0730199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:20 to store for rank: 0 2022-11-23T03:54:46.0730595Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:20 with 2 nodes. 2022-11-23T03:54:46.0730829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:20 to store for rank: 1 2022-11-23T03:54:46.0731225Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:20 with 2 nodes. 2022-11-23T03:54:46.0731455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:21 to store for rank: 0 2022-11-23T03:54:46.0731723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:21 to store for rank: 1 2022-11-23T03:54:46.0732131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:21 with 2 nodes. 2022-11-23T03:54:46.0732528Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:21 with 2 nodes. 2022-11-23T03:54:46.0732759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:22 to store for rank: 0 2022-11-23T03:54:46.0733153Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:22 with 2 nodes. 2022-11-23T03:54:46.0733388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:22 to store for rank: 1 2022-11-23T03:54:46.0733781Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:22 with 2 nodes. 2022-11-23T03:54:46.0734547Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0735323Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0735557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:23 to store for rank: 0 2022-11-23T03:54:46.0735781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:23 to store for rank: 1 2022-11-23T03:54:46.0736176Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:23 with 2 nodes. 2022-11-23T03:54:46.0736577Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:23 with 2 nodes. 2022-11-23T03:54:46.0736812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:24 to store for rank: 0 2022-11-23T03:54:46.0737210Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:24 with 2 nodes. 2022-11-23T03:54:46.0737432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:24 to store for rank: 1 2022-11-23T03:54:46.0737809Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:24 with 2 nodes. 2022-11-23T03:54:46.0738044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:25 to store for rank: 0 2022-11-23T03:54:46.0738266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:25 to store for rank: 1 2022-11-23T03:54:46.0738665Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:25 with 2 nodes. 2022-11-23T03:54:46.0739134Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:25 with 2 nodes. 2022-11-23T03:54:46.0739239Z dist init r=1, world=2 2022-11-23T03:54:46.0739347Z dist init r=0, world=2 2022-11-23T03:54:46.0739451Z ok (8.434s) 2022-11-23T03:54:46.0739776Z test_mixture_of_experts_offload_false_none_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43791 2022-11-23T03:54:46.0739986Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43792 2022-11-23T03:54:46.0740370Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.0740545Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.0740978Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.0741166Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.0741396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.0741784Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.0741951Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.0742341Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.0742525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.0742759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.0743158Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.0743567Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.0743834Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.0744117Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.0744332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.0744551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.0745612Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.0745722Z warnings.warn( 2022-11-23T03:54:46.0745957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T03:54:46.0747014Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.0747123Z warnings.warn( 2022-11-23T03:54:46.0747358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T03:54:46.0747760Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:54:46.0748227Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:54:46.0748456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T03:54:46.0748676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T03:54:46.0749070Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T03:54:46.0749469Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T03:54:46.0750286Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0750528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T03:54:46.0750758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T03:54:46.0751156Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T03:54:46.0751563Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T03:54:46.0752329Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0752559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1 2022-11-23T03:54:46.0752967Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 2 nodes. 2022-11-23T03:54:46.0753195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0 2022-11-23T03:54:46.0753586Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 2 nodes. 2022-11-23T03:54:46.0753816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1 2022-11-23T03:54:46.0754042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0 2022-11-23T03:54:46.0754435Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 2 nodes. 2022-11-23T03:54:46.0754847Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 2 nodes. 2022-11-23T03:54:46.0755612Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0755838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1 2022-11-23T03:54:46.0756072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0 2022-11-23T03:54:46.0756470Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 2 nodes. 2022-11-23T03:54:46.0756866Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 2 nodes. 2022-11-23T03:54:46.0757105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 0 2022-11-23T03:54:46.0758284Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py:1255: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:314.) 2022-11-23T03:54:46.0758477Z _ext_post_unflatten_transform(subtensor.view(shape), param_extension) 2022-11-23T03:54:46.0758711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 1 2022-11-23T03:54:46.0759111Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:8 with 2 nodes. 2022-11-23T03:54:46.0759549Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:8 with 2 nodes. 2022-11-23T03:54:46.0759781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 1 2022-11-23T03:54:46.0760007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 0 2022-11-23T03:54:46.0760405Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:9 with 2 nodes. 2022-11-23T03:54:46.0760783Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:9 with 2 nodes. 2022-11-23T03:54:46.0761539Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0761787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 1 2022-11-23T03:54:46.0762000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 0 2022-11-23T03:54:46.0762396Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:10 with 2 nodes. 2022-11-23T03:54:46.0762789Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:10 with 2 nodes. 2022-11-23T03:54:46.0763552Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0763792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 1 2022-11-23T03:54:46.0764018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 0 2022-11-23T03:54:46.0764415Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:11 with 2 nodes. 2022-11-23T03:54:46.0764812Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:11 with 2 nodes. 2022-11-23T03:54:46.0765047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 1 2022-11-23T03:54:46.0765275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 0 2022-11-23T03:54:46.0765672Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:12 with 2 nodes. 2022-11-23T03:54:46.0766064Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:12 with 2 nodes. 2022-11-23T03:54:46.0766839Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0767160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 1 2022-11-23T03:54:46.0767386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 0 2022-11-23T03:54:46.0767924Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:13 with 2 nodes. 2022-11-23T03:54:46.0768528Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:13 with 2 nodes. 2022-11-23T03:54:46.0769363Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0769606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 1 2022-11-23T03:54:46.0769833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 0 2022-11-23T03:54:46.0770237Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:14 with 2 nodes. 2022-11-23T03:54:46.0770634Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:14 with 2 nodes. 2022-11-23T03:54:46.0770857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 1 2022-11-23T03:54:46.0771077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 0 2022-11-23T03:54:46.0771491Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:15 with 2 nodes. 2022-11-23T03:54:46.0771890Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:15 with 2 nodes. 2022-11-23T03:54:46.0772656Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0772883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 1 2022-11-23T03:54:46.0773105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 0 2022-11-23T03:54:46.0773502Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:16 with 2 nodes. 2022-11-23T03:54:46.0773913Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:16 with 2 nodes. 2022-11-23T03:54:46.0774143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 1 2022-11-23T03:54:46.0774381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 0 2022-11-23T03:54:46.0774780Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:17 with 2 nodes. 2022-11-23T03:54:46.0775540Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0775941Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:17 with 2 nodes. 2022-11-23T03:54:46.0776763Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0776990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 1 2022-11-23T03:54:46.0777225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 0 2022-11-23T03:54:46.0777625Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:18 with 2 nodes. 2022-11-23T03:54:46.0778019Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:18 with 2 nodes. 2022-11-23T03:54:46.0778296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 1 2022-11-23T03:54:46.0778523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 0 2022-11-23T03:54:46.0778907Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:19 with 2 nodes. 2022-11-23T03:54:46.0779304Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:19 with 2 nodes. 2022-11-23T03:54:46.0780063Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0780289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:20 to store for rank: 1 2022-11-23T03:54:46.0780517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:20 to store for rank: 0 2022-11-23T03:54:46.0780914Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:20 with 2 nodes. 2022-11-23T03:54:46.0781308Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:20 with 2 nodes. 2022-11-23T03:54:46.0781531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:21 to store for rank: 1 2022-11-23T03:54:46.0781754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:21 to store for rank: 0 2022-11-23T03:54:46.0782149Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:21 with 2 nodes. 2022-11-23T03:54:46.0782543Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:21 with 2 nodes. 2022-11-23T03:54:46.0782765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:22 to store for rank: 1 2022-11-23T03:54:46.0782988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:22 to store for rank: 0 2022-11-23T03:54:46.0783382Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:22 with 2 nodes. 2022-11-23T03:54:46.0783776Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:22 with 2 nodes. 2022-11-23T03:54:46.0784537Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0784761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:23 to store for rank: 1 2022-11-23T03:54:46.0784988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:23 to store for rank: 0 2022-11-23T03:54:46.0785440Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:23 with 2 nodes. 2022-11-23T03:54:46.0785834Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:23 with 2 nodes. 2022-11-23T03:54:46.0786598Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0786825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:24 to store for rank: 1 2022-11-23T03:54:46.0787053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:24 to store for rank: 0 2022-11-23T03:54:46.0787492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:24 with 2 nodes. 2022-11-23T03:54:46.0787895Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:24 with 2 nodes. 2022-11-23T03:54:46.0788115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:25 to store for rank: 1 2022-11-23T03:54:46.0788337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:25 to store for rank: 0 2022-11-23T03:54:46.0788733Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:25 with 2 nodes. 2022-11-23T03:54:46.0789125Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:25 with 2 nodes. 2022-11-23T03:54:46.0789229Z dist init r=0, world=2 2022-11-23T03:54:46.0789333Z dist init r=1, world=2 2022-11-23T03:54:46.0789429Z ok (8.835s) 2022-11-23T03:54:46.0789769Z test_mixture_of_experts_offload_false_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44184 2022-11-23T03:54:46.0789969Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44185 2022-11-23T03:54:46.0790345Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.0790513Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.0790897Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.0791081Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.0791310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.0791683Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.0791858Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.0792245Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.0792425Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.0792652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.0793049Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.0793441Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.0793722Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.0793998Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.0794221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.0794500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.0795567Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.0795672Z warnings.warn( 2022-11-23T03:54:46.0795896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T03:54:46.0796995Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.0797108Z warnings.warn( 2022-11-23T03:54:46.0797338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T03:54:46.0797738Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:54:46.0798133Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:54:46.0798356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T03:54:46.0798586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T03:54:46.0798986Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T03:54:46.0799383Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T03:54:46.0800145Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0800374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T03:54:46.0800600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T03:54:46.0800981Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T03:54:46.0801378Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T03:54:46.0802140Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0802369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0 2022-11-23T03:54:46.0802594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1 2022-11-23T03:54:46.0802989Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 2 nodes. 2022-11-23T03:54:46.0803385Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 2 nodes. 2022-11-23T03:54:46.0803659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0 2022-11-23T03:54:46.0803884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1 2022-11-23T03:54:46.0804283Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 2 nodes. 2022-11-23T03:54:46.0804679Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 2 nodes. 2022-11-23T03:54:46.0805439Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0805721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1 2022-11-23T03:54:46.0806120Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 2 nodes. 2022-11-23T03:54:46.0806346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0 2022-11-23T03:54:46.0806741Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 2 nodes. 2022-11-23T03:54:46.0806963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 0 2022-11-23T03:54:46.0808100Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py:1255: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:314.) 2022-11-23T03:54:46.0808295Z _ext_post_unflatten_transform(subtensor.view(shape), param_extension) 2022-11-23T03:54:46.0808520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 1 2022-11-23T03:54:46.0808916Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:8 with 2 nodes. 2022-11-23T03:54:46.0809307Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:8 with 2 nodes. 2022-11-23T03:54:46.0809529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 0 2022-11-23T03:54:46.0809751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 1 2022-11-23T03:54:46.0810150Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:9 with 2 nodes. 2022-11-23T03:54:46.0810547Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:9 with 2 nodes. 2022-11-23T03:54:46.0811311Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0811539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 0 2022-11-23T03:54:46.0811764Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 1 2022-11-23T03:54:46.0812162Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:10 with 2 nodes. 2022-11-23T03:54:46.0812557Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:10 with 2 nodes. 2022-11-23T03:54:46.0813397Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0813627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 0 2022-11-23T03:54:46.0813851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 1 2022-11-23T03:54:46.0814250Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:11 with 2 nodes. 2022-11-23T03:54:46.0814646Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:11 with 2 nodes. 2022-11-23T03:54:46.0814918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 0 2022-11-23T03:54:46.0815150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 1 2022-11-23T03:54:46.0815546Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:12 with 2 nodes. 2022-11-23T03:54:46.0815939Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:12 with 2 nodes. 2022-11-23T03:54:46.0816689Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0816914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 0 2022-11-23T03:54:46.0817144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 1 2022-11-23T03:54:46.0817542Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:13 with 2 nodes. 2022-11-23T03:54:46.0817936Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:13 with 2 nodes. 2022-11-23T03:54:46.0818698Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0818924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 0 2022-11-23T03:54:46.0819149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 1 2022-11-23T03:54:46.0819533Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:14 with 2 nodes. 2022-11-23T03:54:46.0819927Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:14 with 2 nodes. 2022-11-23T03:54:46.0820153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 1 2022-11-23T03:54:46.0820544Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:15 with 2 nodes. 2022-11-23T03:54:46.0820766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 0 2022-11-23T03:54:46.0821160Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:15 with 2 nodes. 2022-11-23T03:54:46.0821922Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0822199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 1 2022-11-23T03:54:46.0822422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 0 2022-11-23T03:54:46.0822822Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:16 with 2 nodes. 2022-11-23T03:54:46.0823216Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:16 with 2 nodes. 2022-11-23T03:54:46.0823441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 0 2022-11-23T03:54:46.0823872Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:17 with 2 nodes. 2022-11-23T03:54:46.0824109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 1 2022-11-23T03:54:46.0824507Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:17 with 2 nodes. 2022-11-23T03:54:46.0825263Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0826031Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0826263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 0 2022-11-23T03:54:46.0826486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 1 2022-11-23T03:54:46.0826883Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:18 with 2 nodes. 2022-11-23T03:54:46.0827280Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:18 with 2 nodes. 2022-11-23T03:54:46.0827505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 1 2022-11-23T03:54:46.0827900Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:19 with 2 nodes. 2022-11-23T03:54:46.0828123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 0 2022-11-23T03:54:46.0828519Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:19 with 2 nodes. 2022-11-23T03:54:46.0829284Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0829511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:20 to store for rank: 1 2022-11-23T03:54:46.0829731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:20 to store for rank: 0 2022-11-23T03:54:46.0830127Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:20 with 2 nodes. 2022-11-23T03:54:46.0830522Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:20 with 2 nodes. 2022-11-23T03:54:46.0830798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:21 to store for rank: 0 2022-11-23T03:54:46.0831022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:21 to store for rank: 1 2022-11-23T03:54:46.0831418Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:21 with 2 nodes. 2022-11-23T03:54:46.0831811Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:21 with 2 nodes. 2022-11-23T03:54:46.0832033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:22 to store for rank: 1 2022-11-23T03:54:46.0832255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:22 to store for rank: 0 2022-11-23T03:54:46.0832654Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:22 with 2 nodes. 2022-11-23T03:54:46.0833089Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:22 with 2 nodes. 2022-11-23T03:54:46.0833859Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0834085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:23 to store for rank: 0 2022-11-23T03:54:46.0834484Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:23 with 2 nodes. 2022-11-23T03:54:46.0834694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:23 to store for rank: 1 2022-11-23T03:54:46.0835084Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:23 with 2 nodes. 2022-11-23T03:54:46.0835851Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.0836079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:24 to store for rank: 0 2022-11-23T03:54:46.0836303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:24 to store for rank: 1 2022-11-23T03:54:46.0836702Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:24 with 2 nodes. 2022-11-23T03:54:46.0837095Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:24 with 2 nodes. 2022-11-23T03:54:46.0837318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:25 to store for rank: 1 2022-11-23T03:54:46.0837542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:25 to store for rank: 0 2022-11-23T03:54:46.0837941Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:25 with 2 nodes. 2022-11-23T03:54:46.0838336Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:25 with 2 nodes. 2022-11-23T03:54:46.0838439Z dist init r=1, world=2 2022-11-23T03:54:46.0838541Z dist init r=0, world=2 2022-11-23T03:54:46.0838636Z ok (8.639s) 2022-11-23T03:54:46.0838961Z test_mixture_of_experts_offload_true_no_shard_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44577 2022-11-23T03:54:46.0839167Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44578 2022-11-23T03:54:46.0839544Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.0839772Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.0840164Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.0840345Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.0840568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.0840945Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.0841112Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.0841495Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.0841675Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.0841929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.0842333Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.0842724Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.0843004Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.0843281Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.0843499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.0843716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.0844778Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.0844884Z warnings.warn( 2022-11-23T03:54:46.0845114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T03:54:46.0846178Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.0846287Z warnings.warn( 2022-11-23T03:54:46.0846517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T03:54:46.0846916Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:54:46.0847317Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:54:46.0847444Z File "", line 1, in 2022-11-23T03:54:46.0847647Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0847972Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0848343Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0848486Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0848692Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0848789Z self.run() 2022-11-23T03:54:46.0849067Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0849205Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0849576Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0849690Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0850062Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0850179Z getattr(self, test_name)() 2022-11-23T03:54:46.0850549Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0850640Z fn() 2022-11-23T03:54:46.0851016Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0851133Z test(self, **param_kwargs) 2022-11-23T03:54:46.0851556Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0851682Z return func(*args, **kwargs) 2022-11-23T03:54:46.0851916Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.0852020Z self.run_subtests( 2022-11-23T03:54:46.0852386Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0852540Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0852914Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0853056Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0853445Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0853555Z output = model(*input) 2022-11-23T03:54:46.0853896Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0854020Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0854410Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0854576Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0854953Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0855068Z _lazy_init(state, module) 2022-11-23T03:54:46.0855429Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0855563Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0855911Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0856032Z return func(*args, **kwargs) 2022-11-23T03:54:46.0856430Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0856523Z p_assert( 2022-11-23T03:54:46.0856863Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0856980Z traceback.print_stack() 2022-11-23T03:54:46.0857204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T03:54:46.0857329Z File "", line 1, in 2022-11-23T03:54:46.0857527Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0857662Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0857854Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0857982Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0858187Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0858472Z self.run() 2022-11-23T03:54:46.0858665Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0858804Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0859160Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0859287Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0859658Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0859778Z getattr(self, test_name)() 2022-11-23T03:54:46.0860146Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0860241Z fn() 2022-11-23T03:54:46.0860617Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0860781Z test(self, **param_kwargs) 2022-11-23T03:54:46.0861153Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0861271Z return func(*args, **kwargs) 2022-11-23T03:54:46.0861504Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.0861612Z self.run_subtests( 2022-11-23T03:54:46.0861958Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0862112Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0862484Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0862629Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0863018Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0863131Z output = model(*input) 2022-11-23T03:54:46.0863467Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0863602Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0863987Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0864157Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0864533Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0864648Z _lazy_init(state, module) 2022-11-23T03:54:46.0865010Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0865149Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0865499Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0865622Z return func(*args, **kwargs) 2022-11-23T03:54:46.0866011Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0866105Z p_assert( 2022-11-23T03:54:46.0866432Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0866550Z traceback.print_stack() 2022-11-23T03:54:46.0866776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T03:54:46.0867174Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T03:54:46.0867572Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T03:54:46.0867700Z File "", line 1, in 2022-11-23T03:54:46.0867967Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0868105Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0868298Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0868439Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0868642Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0868739Z self.run() 2022-11-23T03:54:46.0868932Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0869070Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0869422Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0869550Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0869923Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0870090Z getattr(self, test_name)() 2022-11-23T03:54:46.0870451Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0870543Z fn() 2022-11-23T03:54:46.0870917Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0871035Z test(self, **param_kwargs) 2022-11-23T03:54:46.0871403Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0871522Z return func(*args, **kwargs) 2022-11-23T03:54:46.0871756Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.0871862Z self.run_subtests( 2022-11-23T03:54:46.0872223Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0872386Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0872760Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0872903Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0873293Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0873402Z output = model(*input) 2022-11-23T03:54:46.0873740Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0873875Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0874259Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0874430Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0874808Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0874911Z _lazy_init(state, module) 2022-11-23T03:54:46.0875271Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0875407Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0875754Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0875872Z return func(*args, **kwargs) 2022-11-23T03:54:46.0876259Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0876354Z p_assert( 2022-11-23T03:54:46.0876695Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0876811Z traceback.print_stack() 2022-11-23T03:54:46.0877037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T03:54:46.0877236Z File "", line 1, in 2022-11-23T03:54:46.0877436Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0877574Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0877765Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0877907Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0878109Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0878208Z self.run() 2022-11-23T03:54:46.0878386Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0878523Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0878875Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0879001Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0879418Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0879538Z getattr(self, test_name)() 2022-11-23T03:54:46.0879907Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0879998Z fn() 2022-11-23T03:54:46.0880372Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0880489Z test(self, **param_kwargs) 2022-11-23T03:54:46.0880851Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0880969Z return func(*args, **kwargs) 2022-11-23T03:54:46.0881203Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.0881311Z self.run_subtests( 2022-11-23T03:54:46.0881677Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0881833Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0882204Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0882334Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0882717Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0882829Z output = model(*input) 2022-11-23T03:54:46.0883162Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0883297Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0883685Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0883856Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0884232Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0884349Z _lazy_init(state, module) 2022-11-23T03:54:46.0884710Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0884850Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0885192Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0885313Z return func(*args, **kwargs) 2022-11-23T03:54:46.0885700Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0885794Z p_assert( 2022-11-23T03:54:46.0886136Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0886317Z traceback.print_stack() 2022-11-23T03:54:46.0886544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T03:54:46.0886944Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T03:54:46.0887321Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T03:54:46.0887444Z File "", line 1, in 2022-11-23T03:54:46.0887643Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0887852Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0888043Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0888182Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0888385Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0888548Z self.run() 2022-11-23T03:54:46.0888743Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0888881Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0889234Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0889360Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0889729Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0889847Z getattr(self, test_name)() 2022-11-23T03:54:46.0890213Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0890305Z fn() 2022-11-23T03:54:46.0890681Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0890784Z test(self, **param_kwargs) 2022-11-23T03:54:46.0891152Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0891270Z return func(*args, **kwargs) 2022-11-23T03:54:46.0891503Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.0891608Z self.run_subtests( 2022-11-23T03:54:46.0891966Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0892117Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0892488Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0892632Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0893014Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0893134Z output = model(*input) 2022-11-23T03:54:46.0893466Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0893597Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0893986Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0894154Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0894529Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0894644Z _lazy_init(state, module) 2022-11-23T03:54:46.0895001Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0895134Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0895465Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0895659Z return func(*args, **kwargs) 2022-11-23T03:54:46.0896047Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0896140Z p_assert( 2022-11-23T03:54:46.0896481Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0896595Z traceback.print_stack() 2022-11-23T03:54:46.0896826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1 2022-11-23T03:54:46.0896948Z File "", line 1, in 2022-11-23T03:54:46.0897145Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0897281Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0897472Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0897614Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0897865Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0897964Z self.run() 2022-11-23T03:54:46.0898158Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0898294Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0898630Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0898759Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0899131Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0899249Z getattr(self, test_name)() 2022-11-23T03:54:46.0899617Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0899714Z fn() 2022-11-23T03:54:46.0900089Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0900213Z test(self, **param_kwargs) 2022-11-23T03:54:46.0900580Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0900698Z return func(*args, **kwargs) 2022-11-23T03:54:46.0900929Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.0901034Z self.run_subtests( 2022-11-23T03:54:46.0901392Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0901550Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0901920Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0902063Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0902454Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0902565Z output = model(*input) 2022-11-23T03:54:46.0902883Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0903017Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0903398Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0903566Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0903940Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0904054Z _lazy_init(state, module) 2022-11-23T03:54:46.0904415Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0904554Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0904977Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0905096Z return func(*args, **kwargs) 2022-11-23T03:54:46.0905487Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0905581Z p_assert( 2022-11-23T03:54:46.0905921Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0906039Z traceback.print_stack() 2022-11-23T03:54:46.0906265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0 2022-11-23T03:54:46.0906662Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 2 nodes. 2022-11-23T03:54:46.0907054Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 2 nodes. 2022-11-23T03:54:46.0907227Z File "", line 1, in 2022-11-23T03:54:46.0907433Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0907552Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0907745Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0907887Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0908089Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0908184Z self.run() 2022-11-23T03:54:46.0908379Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0908517Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0908868Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0908996Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0909371Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0909489Z getattr(self, test_name)() 2022-11-23T03:54:46.0909856Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0909948Z fn() 2022-11-23T03:54:46.0910320Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0910437Z test(self, **param_kwargs) 2022-11-23T03:54:46.0910803Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0910921Z return func(*args, **kwargs) 2022-11-23T03:54:46.0911141Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.0911249Z self.run_subtests( 2022-11-23T03:54:46.0911611Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0911768Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0912139Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0912286Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0912669Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0912780Z output = model(*input) 2022-11-23T03:54:46.0913113Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0913247Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0913629Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0913794Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0914238Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0914355Z _lazy_init(state, module) 2022-11-23T03:54:46.0914712Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0914847Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0915194Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0915311Z return func(*args, **kwargs) 2022-11-23T03:54:46.0915696Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0915776Z p_assert( 2022-11-23T03:54:46.0916117Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0916234Z traceback.print_stack() 2022-11-23T03:54:46.0916514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0 2022-11-23T03:54:46.0916640Z File "", line 1, in 2022-11-23T03:54:46.0916839Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0916975Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0917168Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0917307Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0917513Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0917609Z self.run() 2022-11-23T03:54:46.0917803Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0917940Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0918291Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0918426Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0918795Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0918912Z getattr(self, test_name)() 2022-11-23T03:54:46.0919265Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0919357Z fn() 2022-11-23T03:54:46.0919731Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0919849Z test(self, **param_kwargs) 2022-11-23T03:54:46.0920214Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0920332Z return func(*args, **kwargs) 2022-11-23T03:54:46.0920565Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.0920678Z self.run_subtests( 2022-11-23T03:54:46.0921041Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0921195Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0921568Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0921715Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0922096Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0922207Z output = model(*input) 2022-11-23T03:54:46.0922540Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0922675Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0923061Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0923290Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0923654Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0923769Z _lazy_init(state, module) 2022-11-23T03:54:46.0924127Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0924260Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0924606Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0924724Z return func(*args, **kwargs) 2022-11-23T03:54:46.0925109Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0925203Z p_assert( 2022-11-23T03:54:46.0925589Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0925713Z traceback.print_stack() 2022-11-23T03:54:46.0925944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1 2022-11-23T03:54:46.0926341Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 2 nodes. 2022-11-23T03:54:46.0926737Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 2 nodes. 2022-11-23T03:54:46.0926862Z File "", line 1, in 2022-11-23T03:54:46.0927062Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0927196Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0927388Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0927529Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0927784Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0927882Z self.run() 2022-11-23T03:54:46.0928079Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0928215Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0928566Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0928697Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0929065Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0929183Z getattr(self, test_name)() 2022-11-23T03:54:46.0929552Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0929643Z fn() 2022-11-23T03:54:46.0930016Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0930135Z test(self, **param_kwargs) 2022-11-23T03:54:46.0930499Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0930615Z return func(*args, **kwargs) 2022-11-23T03:54:46.0930850Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.0930955Z self.run_subtests( 2022-11-23T03:54:46.0931315Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0931455Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0931827Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0931972Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0932356Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0932537Z output = model(*input) 2022-11-23T03:54:46.0932876Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0933009Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0933392Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0933557Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0933934Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0934048Z _lazy_init(state, module) 2022-11-23T03:54:46.0934408Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0934540Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0934937Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0935057Z return func(*args, **kwargs) 2022-11-23T03:54:46.0935447Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0935540Z p_assert( 2022-11-23T03:54:46.0935881Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0935999Z traceback.print_stack() 2022-11-23T03:54:46.0936210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1 2022-11-23T03:54:46.0936332Z File "", line 1, in 2022-11-23T03:54:46.0936528Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0936662Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0936856Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0937003Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0937209Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0937305Z self.run() 2022-11-23T03:54:46.0937497Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0937635Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0937981Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0938107Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0938475Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0938593Z getattr(self, test_name)() 2022-11-23T03:54:46.0938956Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0939047Z fn() 2022-11-23T03:54:46.0939413Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0939532Z test(self, **param_kwargs) 2022-11-23T03:54:46.0939897Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0940015Z return func(*args, **kwargs) 2022-11-23T03:54:46.0940246Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.0940350Z self.run_subtests( 2022-11-23T03:54:46.0940709Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0940862Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0941232Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0941446Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0941830Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0941941Z output = model(*input) 2022-11-23T03:54:46.0942274Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0942411Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0942795Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0942958Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0943330Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0943447Z _lazy_init(state, module) 2022-11-23T03:54:46.0943888Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0944015Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0944368Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0944486Z return func(*args, **kwargs) 2022-11-23T03:54:46.0944871Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0944964Z p_assert( 2022-11-23T03:54:46.0945306Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0945421Z traceback.print_stack() 2022-11-23T03:54:46.0945648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0 2022-11-23T03:54:46.0946046Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 2 nodes. 2022-11-23T03:54:46.0946445Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 2 nodes. 2022-11-23T03:54:46.0946571Z File "", line 1, in 2022-11-23T03:54:46.0946772Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0946905Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0947100Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0947246Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0947446Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0947543Z self.run() 2022-11-23T03:54:46.0947735Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0947858Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0948205Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0948339Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0948710Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0948828Z getattr(self, test_name)() 2022-11-23T03:54:46.0949191Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0949284Z fn() 2022-11-23T03:54:46.0949655Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0949773Z test(self, **param_kwargs) 2022-11-23T03:54:46.0950137Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0950253Z return func(*args, **kwargs) 2022-11-23T03:54:46.0950486Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.0950666Z self.run_subtests( 2022-11-23T03:54:46.0951030Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0951183Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0951552Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0951699Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0952084Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0952180Z output = model(*input) 2022-11-23T03:54:46.0952513Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0952650Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0953031Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0953243Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0953621Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0953736Z _lazy_init(state, module) 2022-11-23T03:54:46.0954096Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0954232Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0954577Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0954695Z return func(*args, **kwargs) 2022-11-23T03:54:46.0955083Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0955178Z p_assert( 2022-11-23T03:54:46.0955521Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0955640Z traceback.print_stack() 2022-11-23T03:54:46.0955868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 0 2022-11-23T03:54:46.0955994Z File "", line 1, in 2022-11-23T03:54:46.0956193Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0956313Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0956505Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0956647Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0956850Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0956946Z self.run() 2022-11-23T03:54:46.0957141Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0957282Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0957635Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0957761Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0958137Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0958252Z getattr(self, test_name)() 2022-11-23T03:54:46.0958619Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0958710Z fn() 2022-11-23T03:54:46.0959081Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0959202Z test(self, **param_kwargs) 2022-11-23T03:54:46.0959564Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0959681Z return func(*args, **kwargs) 2022-11-23T03:54:46.0959903Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.0960078Z self.run_subtests( 2022-11-23T03:54:46.0960439Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0960594Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0960964Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0961107Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0961487Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0961597Z output = model(*input) 2022-11-23T03:54:46.0961933Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0962069Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0962500Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0962672Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0963049Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0963164Z _lazy_init(state, module) 2022-11-23T03:54:46.0963523Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0963659Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0964004Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0964122Z return func(*args, **kwargs) 2022-11-23T03:54:46.0964492Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0964590Z p_assert( 2022-11-23T03:54:46.0964934Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0965049Z traceback.print_stack() 2022-11-23T03:54:46.0965278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 1 2022-11-23T03:54:46.0965678Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:8 with 2 nodes. 2022-11-23T03:54:46.0966071Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:8 with 2 nodes. 2022-11-23T03:54:46.0966193Z File "", line 1, in 2022-11-23T03:54:46.0966395Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0966528Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0966722Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0966873Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0967076Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0967177Z self.run() 2022-11-23T03:54:46.0967366Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0967504Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0968006Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0968136Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0968495Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0968613Z getattr(self, test_name)() 2022-11-23T03:54:46.0968982Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0969073Z fn() 2022-11-23T03:54:46.0969452Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0969650Z test(self, **param_kwargs) 2022-11-23T03:54:46.0970018Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0970136Z return func(*args, **kwargs) 2022-11-23T03:54:46.0970371Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.0970476Z self.run_subtests( 2022-11-23T03:54:46.0970834Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0970987Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0971357Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0971500Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0971937Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0972051Z output = model(*input) 2022-11-23T03:54:46.0972389Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0972525Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0972894Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0973061Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0973436Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0973550Z _lazy_init(state, module) 2022-11-23T03:54:46.0973906Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0974047Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0974392Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0974511Z return func(*args, **kwargs) 2022-11-23T03:54:46.0974897Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0974990Z p_assert( 2022-11-23T03:54:46.0975329Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0975445Z traceback.print_stack() 2022-11-23T03:54:46.0975670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 1 2022-11-23T03:54:46.0975792Z File "", line 1, in 2022-11-23T03:54:46.0975994Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0976126Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0976324Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0976452Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0976655Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0976751Z self.run() 2022-11-23T03:54:46.0976944Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0977081Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0977425Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0977552Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0977924Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0978042Z getattr(self, test_name)() 2022-11-23T03:54:46.0978410Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0978563Z fn() 2022-11-23T03:54:46.0978940Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0979057Z test(self, **param_kwargs) 2022-11-23T03:54:46.0979422Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0979539Z return func(*args, **kwargs) 2022-11-23T03:54:46.0979769Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.0979877Z self.run_subtests( 2022-11-23T03:54:46.0980236Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0980376Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0980786Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0980941Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0981324Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0981434Z output = model(*input) 2022-11-23T03:54:46.0981766Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0981900Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0982282Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0982447Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0982821Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0982936Z _lazy_init(state, module) 2022-11-23T03:54:46.0983299Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0983433Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0983777Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0983894Z return func(*args, **kwargs) 2022-11-23T03:54:46.0984284Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0984379Z p_assert( 2022-11-23T03:54:46.0984720Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0984822Z traceback.print_stack() 2022-11-23T03:54:46.0985049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 0 2022-11-23T03:54:46.0985444Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:9 with 2 nodes. 2022-11-23T03:54:46.0985839Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:9 with 2 nodes. 2022-11-23T03:54:46.0985966Z File "", line 1, in 2022-11-23T03:54:46.0986166Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0986302Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0986492Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0986633Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0986833Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0986930Z self.run() 2022-11-23T03:54:46.0987123Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0987260Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0987606Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0987799Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0988174Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0988290Z getattr(self, test_name)() 2022-11-23T03:54:46.0988654Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0988732Z fn() 2022-11-23T03:54:46.0989104Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0989223Z test(self, **param_kwargs) 2022-11-23T03:54:46.0989588Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0989709Z return func(*args, **kwargs) 2022-11-23T03:54:46.0989986Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.0990099Z self.run_subtests( 2022-11-23T03:54:46.0990460Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.0990616Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.0990985Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.0991130Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.0991511Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.0991624Z output = model(*input) 2022-11-23T03:54:46.0991956Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.0992093Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.0992480Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.0992648Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.0993023Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.0993123Z _lazy_init(state, module) 2022-11-23T03:54:46.0993481Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.0993618Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.0993964Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.0994088Z return func(*args, **kwargs) 2022-11-23T03:54:46.0994472Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.0994568Z p_assert( 2022-11-23T03:54:46.0994914Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.0995031Z traceback.print_stack() 2022-11-23T03:54:46.0995260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 0 2022-11-23T03:54:46.0995384Z File "", line 1, in 2022-11-23T03:54:46.0995585Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.0995721Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.0995915Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.0996058Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.0996261Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.0996359Z self.run() 2022-11-23T03:54:46.0996538Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.0996738Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.0997087Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.0997215Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.0997583Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.0997700Z getattr(self, test_name)() 2022-11-23T03:54:46.0998065Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.0998158Z fn() 2022-11-23T03:54:46.0998530Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.0998647Z test(self, **param_kwargs) 2022-11-23T03:54:46.0999009Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.0999182Z return func(*args, **kwargs) 2022-11-23T03:54:46.0999417Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.0999523Z self.run_subtests( 2022-11-23T03:54:46.0999883Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1000035Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1000406Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1000549Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1000918Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1001029Z output = model(*input) 2022-11-23T03:54:46.1001362Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1001503Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1001883Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1002051Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1002424Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1002541Z _lazy_init(state, module) 2022-11-23T03:54:46.1002900Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1003034Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1003378Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1003496Z return func(*args, **kwargs) 2022-11-23T03:54:46.1003882Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1003980Z p_assert( 2022-11-23T03:54:46.1004321Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1004436Z traceback.print_stack() 2022-11-23T03:54:46.1004665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 1 2022-11-23T03:54:46.1005065Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:10 with 2 nodes. 2022-11-23T03:54:46.1005460Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:10 with 2 nodes. 2022-11-23T03:54:46.1005569Z File "", line 1, in 2022-11-23T03:54:46.1005769Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1005904Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1006164Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1006306Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1006508Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1006607Z self.run() 2022-11-23T03:54:46.1006800Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1006936Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1007286Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1007416Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1007861Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1007978Z getattr(self, test_name)() 2022-11-23T03:54:46.1008347Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1008499Z fn() 2022-11-23T03:54:46.1008878Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1008994Z test(self, **param_kwargs) 2022-11-23T03:54:46.1009346Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1009463Z return func(*args, **kwargs) 2022-11-23T03:54:46.1009696Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1009801Z self.run_subtests( 2022-11-23T03:54:46.1010161Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1010313Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1010683Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1010832Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1011214Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1011325Z output = model(*input) 2022-11-23T03:54:46.1011659Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1011792Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1012176Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1012342Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1012718Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1012832Z _lazy_init(state, module) 2022-11-23T03:54:46.1013192Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1013333Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1013664Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1013783Z return func(*args, **kwargs) 2022-11-23T03:54:46.1014171Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1014264Z p_assert( 2022-11-23T03:54:46.1014605Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1014721Z traceback.print_stack() 2022-11-23T03:54:46.1014951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 1 2022-11-23T03:54:46.1015073Z File "", line 1, in 2022-11-23T03:54:46.1015273Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1015488Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1015680Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1015820Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1016023Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1016121Z self.run() 2022-11-23T03:54:46.1016314Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1016453Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1016803Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1016915Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1017283Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1017403Z getattr(self, test_name)() 2022-11-23T03:54:46.1017816Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1017914Z fn() 2022-11-23T03:54:46.1018288Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1018407Z test(self, **param_kwargs) 2022-11-23T03:54:46.1018773Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1018891Z return func(*args, **kwargs) 2022-11-23T03:54:46.1019124Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1019229Z self.run_subtests( 2022-11-23T03:54:46.1019588Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1019742Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1020119Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1020263Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1020644Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1020759Z output = model(*input) 2022-11-23T03:54:46.1021092Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1021212Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1021592Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1021759Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1022132Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1022251Z _lazy_init(state, module) 2022-11-23T03:54:46.1022608Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1022738Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1023078Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1023190Z return func(*args, **kwargs) 2022-11-23T03:54:46.1023571Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1023663Z p_assert( 2022-11-23T03:54:46.1023998Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1024108Z traceback.print_stack() 2022-11-23T03:54:46.1024331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 0 2022-11-23T03:54:46.1024727Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:11 with 2 nodes. 2022-11-23T03:54:46.1025181Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:11 with 2 nodes. 2022-11-23T03:54:46.1025299Z File "", line 1, in 2022-11-23T03:54:46.1025492Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1025621Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1025797Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1025934Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1026131Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1026223Z self.run() 2022-11-23T03:54:46.1026412Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1026543Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1026930Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1027055Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1027420Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1027532Z getattr(self, test_name)() 2022-11-23T03:54:46.1027891Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1027977Z fn() 2022-11-23T03:54:46.1028342Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1028453Z test(self, **param_kwargs) 2022-11-23T03:54:46.1028812Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1028924Z return func(*args, **kwargs) 2022-11-23T03:54:46.1029160Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1029251Z self.run_subtests( 2022-11-23T03:54:46.1029606Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1029754Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1030119Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1030261Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1030640Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1030745Z output = model(*input) 2022-11-23T03:54:46.1031071Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1031204Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1031581Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1031740Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1032109Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1032218Z _lazy_init(state, module) 2022-11-23T03:54:46.1032570Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1032700Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1033036Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1033149Z return func(*args, **kwargs) 2022-11-23T03:54:46.1033531Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1033684Z p_assert( 2022-11-23T03:54:46.1034014Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1034125Z traceback.print_stack() 2022-11-23T03:54:46.1034348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 1 2022-11-23T03:54:46.1034464Z File "", line 1, in 2022-11-23T03:54:46.1034657Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1034786Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1034970Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1035106Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1035303Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1035395Z self.run() 2022-11-23T03:54:46.1035625Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1035763Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1036105Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1036225Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1036588Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1036699Z getattr(self, test_name)() 2022-11-23T03:54:46.1037052Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1037140Z fn() 2022-11-23T03:54:46.1037507Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1037619Z test(self, **param_kwargs) 2022-11-23T03:54:46.1037981Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1038096Z return func(*args, **kwargs) 2022-11-23T03:54:46.1038320Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1038422Z self.run_subtests( 2022-11-23T03:54:46.1038773Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1038923Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1039289Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1039427Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1039804Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1039909Z output = model(*input) 2022-11-23T03:54:46.1040238Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1040369Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1040748Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1040909Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1041277Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1041377Z _lazy_init(state, module) 2022-11-23T03:54:46.1041730Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1041860Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1042200Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1042312Z return func(*args, **kwargs) 2022-11-23T03:54:46.1042757Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1042846Z p_assert( 2022-11-23T03:54:46.1043180Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1043291Z traceback.print_stack() 2022-11-23T03:54:46.1043516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 0 2022-11-23T03:54:46.1043912Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:12 with 2 nodes. 2022-11-23T03:54:46.1044301Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:12 with 2 nodes. 2022-11-23T03:54:46.1044420Z File "", line 1, in 2022-11-23T03:54:46.1044616Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1044800Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1044991Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1045128Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1045327Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1045410Z self.run() 2022-11-23T03:54:46.1045598Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1045728Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1046075Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1046195Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1046558Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1046669Z getattr(self, test_name)() 2022-11-23T03:54:46.1047035Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1047124Z fn() 2022-11-23T03:54:46.1047491Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1047602Z test(self, **param_kwargs) 2022-11-23T03:54:46.1048103Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1048217Z return func(*args, **kwargs) 2022-11-23T03:54:46.1048444Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1048543Z self.run_subtests( 2022-11-23T03:54:46.1048898Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1049048Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1049408Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1049550Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1049928Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1050034Z output = model(*input) 2022-11-23T03:54:46.1050361Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1050490Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1050866Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1051029Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1051399Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1051589Z _lazy_init(state, module) 2022-11-23T03:54:46.1051944Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1052072Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1052412Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1052525Z return func(*args, **kwargs) 2022-11-23T03:54:46.1052905Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1052993Z p_assert( 2022-11-23T03:54:46.1053329Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1053439Z traceback.print_stack() 2022-11-23T03:54:46.1053654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 0 2022-11-23T03:54:46.1053771Z File "", line 1, in 2022-11-23T03:54:46.1054019Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1054152Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1054340Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1054476Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1054673Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1054763Z self.run() 2022-11-23T03:54:46.1054949Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1055081Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1055424Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1055545Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1055911Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1056030Z getattr(self, test_name)() 2022-11-23T03:54:46.1056388Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1056573Z fn() 2022-11-23T03:54:46.1056961Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1057066Z test(self, **param_kwargs) 2022-11-23T03:54:46.1057458Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1057588Z return func(*args, **kwargs) 2022-11-23T03:54:46.1057831Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1057946Z self.run_subtests( 2022-11-23T03:54:46.1058307Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1058617Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1059002Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1059156Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1059688Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1059807Z output = model(*input) 2022-11-23T03:54:46.1060152Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1060295Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1060692Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1060901Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1061291Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1061501Z _lazy_init(state, module) 2022-11-23T03:54:46.1061871Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1062013Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1062369Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1062495Z return func(*args, **kwargs) 2022-11-23T03:54:46.1062890Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1063021Z p_assert( 2022-11-23T03:54:46.1063372Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1063474Z traceback.print_stack() 2022-11-23T03:54:46.1063772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 1 2022-11-23T03:54:46.1064200Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:13 with 2 nodes. 2022-11-23T03:54:46.1064602Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:13 with 2 nodes. 2022-11-23T03:54:46.1064731Z File "", line 1, in 2022-11-23T03:54:46.1064937Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1065082Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1065310Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1065474Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1065685Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1065789Z self.run() 2022-11-23T03:54:46.1065992Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1066142Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1066502Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1066635Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1067044Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1067294Z getattr(self, test_name)() 2022-11-23T03:54:46.1067668Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1067747Z fn() 2022-11-23T03:54:46.1068131Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1068257Z test(self, **param_kwargs) 2022-11-23T03:54:46.1068635Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1068765Z return func(*args, **kwargs) 2022-11-23T03:54:46.1069012Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1069156Z self.run_subtests( 2022-11-23T03:54:46.1069529Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1069692Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1070071Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1070222Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1070616Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1070735Z output = model(*input) 2022-11-23T03:54:46.1071095Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1071331Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1071734Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1071908Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1072292Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1072394Z _lazy_init(state, module) 2022-11-23T03:54:46.1072759Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1072901Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1073270Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1073398Z return func(*args, **kwargs) 2022-11-23T03:54:46.1073871Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1073979Z p_assert( 2022-11-23T03:54:46.1074336Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1074461Z traceback.print_stack() 2022-11-23T03:54:46.1074697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 1 2022-11-23T03:54:46.1074842Z File "", line 1, in 2022-11-23T03:54:46.1075048Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1075193Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1075534Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1075684Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1075894Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1076007Z self.run() 2022-11-23T03:54:46.1076186Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1076333Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1076704Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1076840Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1077217Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1077365Z getattr(self, test_name)() 2022-11-23T03:54:46.1077744Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1077843Z fn() 2022-11-23T03:54:46.1078223Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1078362Z test(self, **param_kwargs) 2022-11-23T03:54:46.1078743Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1078871Z return func(*args, **kwargs) 2022-11-23T03:54:46.1079113Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1079252Z self.run_subtests( 2022-11-23T03:54:46.1079622Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1079782Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1080162Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1080324Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1080695Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1080892Z output = model(*input) 2022-11-23T03:54:46.1081241Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1081403Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1081822Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1081998Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1082383Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1082523Z _lazy_init(state, module) 2022-11-23T03:54:46.1082890Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1083036Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1083562Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1083697Z return func(*args, **kwargs) 2022-11-23T03:54:46.1084121Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1084227Z p_assert( 2022-11-23T03:54:46.1084589Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1084714Z traceback.print_stack() 2022-11-23T03:54:46.1084952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 0 2022-11-23T03:54:46.1085363Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:14 with 2 nodes. 2022-11-23T03:54:46.1085771Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:14 with 2 nodes. 2022-11-23T03:54:46.1085983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 1 2022-11-23T03:54:46.1086226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 0 2022-11-23T03:54:46.1086655Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:15 with 2 nodes. 2022-11-23T03:54:46.1087068Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:15 with 2 nodes. 2022-11-23T03:54:46.1087308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 1 2022-11-23T03:54:46.1087543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 0 2022-11-23T03:54:46.1088013Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:16 with 2 nodes. 2022-11-23T03:54:46.1088419Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:16 with 2 nodes. 2022-11-23T03:54:46.1088656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 1 2022-11-23T03:54:46.1089062Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:17 with 2 nodes. 2022-11-23T03:54:46.1089315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 0 2022-11-23T03:54:46.1089729Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:17 with 2 nodes. 2022-11-23T03:54:46.1089965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 1 2022-11-23T03:54:46.1090199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 0 2022-11-23T03:54:46.1090605Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:18 with 2 nodes. 2022-11-23T03:54:46.1091088Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:18 with 2 nodes. 2022-11-23T03:54:46.1091872Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1092644Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1093506Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1094286Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1095059Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1095826Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1096596Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1097467Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1097706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 0 2022-11-23T03:54:46.1097954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 1 2022-11-23T03:54:46.1098392Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:19 with 2 nodes. 2022-11-23T03:54:46.1098799Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:19 with 2 nodes. 2022-11-23T03:54:46.1099031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:20 to store for rank: 1 2022-11-23T03:54:46.1099262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:20 to store for rank: 0 2022-11-23T03:54:46.1099669Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:20 with 2 nodes. 2022-11-23T03:54:46.1100068Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:20 with 2 nodes. 2022-11-23T03:54:46.1100313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:21 to store for rank: 1 2022-11-23T03:54:46.1100787Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:21 with 2 nodes. 2022-11-23T03:54:46.1101044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:21 to store for rank: 0 2022-11-23T03:54:46.1101449Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:21 with 2 nodes. 2022-11-23T03:54:46.1101679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:22 to store for rank: 0 2022-11-23T03:54:46.1101909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:22 to store for rank: 1 2022-11-23T03:54:46.1102314Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:22 with 2 nodes. 2022-11-23T03:54:46.1102726Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:22 with 2 nodes. 2022-11-23T03:54:46.1103562Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1104343Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1104604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:23 to store for rank: 1 2022-11-23T03:54:46.1105009Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:23 with 2 nodes. 2022-11-23T03:54:46.1105252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:23 to store for rank: 0 2022-11-23T03:54:46.1105664Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:23 with 2 nodes. 2022-11-23T03:54:46.1105908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:24 to store for rank: 1 2022-11-23T03:54:46.1106320Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:24 with 2 nodes. 2022-11-23T03:54:46.1106551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:24 to store for rank: 0 2022-11-23T03:54:46.1106951Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:24 with 2 nodes. 2022-11-23T03:54:46.1107203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:25 to store for rank: 0 2022-11-23T03:54:46.1107435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:25 to store for rank: 1 2022-11-23T03:54:46.1107845Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:25 with 2 nodes. 2022-11-23T03:54:46.1108366Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:25 with 2 nodes. 2022-11-23T03:54:46.1108578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:26 to store for rank: 1 2022-11-23T03:54:46.1108821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:26 to store for rank: 0 2022-11-23T03:54:46.1109230Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:26 with 2 nodes. 2022-11-23T03:54:46.1109629Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:26 with 2 nodes. 2022-11-23T03:54:46.1110402Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1111253Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1111489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:27 to store for rank: 1 2022-11-23T03:54:46.1111723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:27 to store for rank: 0 2022-11-23T03:54:46.1112142Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:27 with 2 nodes. 2022-11-23T03:54:46.1112593Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:27 with 2 nodes. 2022-11-23T03:54:46.1112837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:28 to store for rank: 1 2022-11-23T03:54:46.1113068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:28 to store for rank: 0 2022-11-23T03:54:46.1113479Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:28 with 2 nodes. 2022-11-23T03:54:46.1113901Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:28 with 2 nodes. 2022-11-23T03:54:46.1114132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:29 to store for rank: 1 2022-11-23T03:54:46.1114549Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:29 with 2 nodes. 2022-11-23T03:54:46.1114786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:29 to store for rank: 0 2022-11-23T03:54:46.1115191Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:29 with 2 nodes. 2022-11-23T03:54:46.1115424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:30 to store for rank: 1 2022-11-23T03:54:46.1115826Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:30 with 2 nodes. 2022-11-23T03:54:46.1116059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:30 to store for rank: 0 2022-11-23T03:54:46.1116482Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:30 with 2 nodes. 2022-11-23T03:54:46.1117265Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1118038Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1118278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:31 to store for rank: 1 2022-11-23T03:54:46.1118509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:31 to store for rank: 0 2022-11-23T03:54:46.1118913Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:31 with 2 nodes. 2022-11-23T03:54:46.1119316Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:31 with 2 nodes. 2022-11-23T03:54:46.1119622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:32 to store for rank: 1 2022-11-23T03:54:46.1119890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:32 to store for rank: 0 2022-11-23T03:54:46.1120415Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:32 with 2 nodes. 2022-11-23T03:54:46.1120816Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:32 with 2 nodes. 2022-11-23T03:54:46.1121048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:33 to store for rank: 0 2022-11-23T03:54:46.1121277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:33 to store for rank: 1 2022-11-23T03:54:46.1121681Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:33 with 2 nodes. 2022-11-23T03:54:46.1122137Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:33 with 2 nodes. 2022-11-23T03:54:46.1122393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:34 to store for rank: 1 2022-11-23T03:54:46.1122650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:34 to store for rank: 0 2022-11-23T03:54:46.1123040Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:34 with 2 nodes. 2022-11-23T03:54:46.1123440Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:34 with 2 nodes. 2022-11-23T03:54:46.1124208Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1125014Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1125783Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1126541Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1172055Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1173020Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1173784Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1174872Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1175094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:35 to store for rank: 0 2022-11-23T03:54:46.1175312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:35 to store for rank: 1 2022-11-23T03:54:46.1175709Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:35 with 2 nodes. 2022-11-23T03:54:46.1176097Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:35 with 2 nodes. 2022-11-23T03:54:46.1176316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:36 to store for rank: 1 2022-11-23T03:54:46.1176640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:36 to store for rank: 0 2022-11-23T03:54:46.1177035Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:36 with 2 nodes. 2022-11-23T03:54:46.1177423Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:36 with 2 nodes. 2022-11-23T03:54:46.1177636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:37 to store for rank: 1 2022-11-23T03:54:46.1178021Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:37 with 2 nodes. 2022-11-23T03:54:46.1178238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:37 to store for rank: 0 2022-11-23T03:54:46.1178620Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:37 with 2 nodes. 2022-11-23T03:54:46.1178715Z dist init r=1, world=2 2022-11-23T03:54:46.1179030Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.1179331Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.1179636Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.1179934Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.1180228Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.1180523Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.1180818Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.1181105Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.1181394Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.1181683Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.1181976Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.1182317Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.1182412Z dist init r=0, world=2 2022-11-23T03:54:46.1182709Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.1183002Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.1183295Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.1183623Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.1183921Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.1184212Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.1184501Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.1184789Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.1185077Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.1185371Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.1185663Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.1185952Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.1186036Z ok (9.142s) 2022-11-23T03:54:46.1186355Z test_mixture_of_experts_offload_true_none_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44994 2022-11-23T03:54:46.1186559Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44995 2022-11-23T03:54:46.1186944Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.1187103Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.1187483Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.1187650Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.1187867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.1188235Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.1188393Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.1188768Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.1188939Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.1189213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.1189607Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.1189991Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.1190258Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.1190526Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.1190734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.1190940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.1192035Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.1192137Z warnings.warn( 2022-11-23T03:54:46.1192351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T03:54:46.1193391Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.1193486Z warnings.warn( 2022-11-23T03:54:46.1193706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T03:54:46.1194093Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:54:46.1194477Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:54:46.1194590Z File "", line 1, in 2022-11-23T03:54:46.1194780Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1194905Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1195088Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1195222Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1195416Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1195502Z self.run() 2022-11-23T03:54:46.1195687Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1195814Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1196151Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1196267Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1196632Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1196738Z getattr(self, test_name)() 2022-11-23T03:54:46.1197097Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1197180Z fn() 2022-11-23T03:54:46.1197545Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1197656Z test(self, **param_kwargs) 2022-11-23T03:54:46.1198073Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1198180Z return func(*args, **kwargs) 2022-11-23T03:54:46.1198403Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1198498Z self.run_subtests( 2022-11-23T03:54:46.1198844Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1198988Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1199349Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1199480Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1199854Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1199956Z output = model(*input) 2022-11-23T03:54:46.1200333Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1200460Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1200839Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1200999Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1201366Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1201472Z _lazy_init(state, module) 2022-11-23T03:54:46.1201820Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1201943Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1202281Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1202395Z return func(*args, **kwargs) 2022-11-23T03:54:46.1202774Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1202858Z p_assert( 2022-11-23T03:54:46.1203191Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1203298Z traceback.print_stack() 2022-11-23T03:54:46.1203512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T03:54:46.1203620Z File "", line 1, in 2022-11-23T03:54:46.1203809Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1203934Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1204116Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1204248Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1204448Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1204534Z self.run() 2022-11-23T03:54:46.1204717Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1204843Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1205180Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1205296Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1205661Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1205775Z getattr(self, test_name)() 2022-11-23T03:54:46.1206142Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1206232Z fn() 2022-11-23T03:54:46.1206608Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1206776Z test(self, **param_kwargs) 2022-11-23T03:54:46.1207130Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1207246Z return func(*args, **kwargs) 2022-11-23T03:54:46.1207477Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1207583Z self.run_subtests( 2022-11-23T03:54:46.1208005Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1208157Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1208531Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1208673Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1209111Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1209229Z output = model(*input) 2022-11-23T03:54:46.1209562Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1209695Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1210078Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1210241Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1210616Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1210729Z _lazy_init(state, module) 2022-11-23T03:54:46.1211085Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1211217Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1211553Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1211670Z return func(*args, **kwargs) 2022-11-23T03:54:46.1212056Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1212149Z p_assert( 2022-11-23T03:54:46.1212489Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1212604Z traceback.print_stack() 2022-11-23T03:54:46.1212832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T03:54:46.1213225Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T03:54:46.1213618Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T03:54:46.1213746Z File "", line 1, in 2022-11-23T03:54:46.1213946Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1214077Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1214270Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1214410Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1214612Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1214706Z self.run() 2022-11-23T03:54:46.1214897Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1215034Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1215365Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1215490Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1215866Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1216044Z getattr(self, test_name)() 2022-11-23T03:54:46.1216413Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1216502Z fn() 2022-11-23T03:54:46.1216876Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1216991Z test(self, **param_kwargs) 2022-11-23T03:54:46.1217357Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1217471Z return func(*args, **kwargs) 2022-11-23T03:54:46.1217703Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1217805Z self.run_subtests( 2022-11-23T03:54:46.1218211Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1218369Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1218743Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1218885Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1219265Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1219376Z output = model(*input) 2022-11-23T03:54:46.1219694Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1219829Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1220213Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1220379Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1220759Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1220872Z _lazy_init(state, module) 2022-11-23T03:54:46.1221230Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1221362Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1221704Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1221821Z return func(*args, **kwargs) 2022-11-23T03:54:46.1222207Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1222299Z p_assert( 2022-11-23T03:54:46.1222638Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1222754Z traceback.print_stack() 2022-11-23T03:54:46.1222982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T03:54:46.1223102Z File "", line 1, in 2022-11-23T03:54:46.1223301Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1223435Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1223614Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1223756Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1223959Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1224054Z self.run() 2022-11-23T03:54:46.1224246Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1224381Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1224727Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1224913Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1225288Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1225403Z getattr(self, test_name)() 2022-11-23T03:54:46.1225766Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1225855Z fn() 2022-11-23T03:54:46.1226227Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1226342Z test(self, **param_kwargs) 2022-11-23T03:54:46.1226705Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1226817Z return func(*args, **kwargs) 2022-11-23T03:54:46.1227050Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1227145Z self.run_subtests( 2022-11-23T03:54:46.1227551Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1227706Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1228079Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1228222Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1228602Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1228708Z output = model(*input) 2022-11-23T03:54:46.1229039Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1229167Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1229551Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1229725Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1230095Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1230210Z _lazy_init(state, module) 2022-11-23T03:54:46.1230565Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1230695Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1231038Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1231152Z return func(*args, **kwargs) 2022-11-23T03:54:46.1231536Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1231616Z p_assert( 2022-11-23T03:54:46.1231958Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1232078Z traceback.print_stack() 2022-11-23T03:54:46.1232305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T03:54:46.1232698Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T03:54:46.1233085Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T03:54:46.1233205Z File "", line 1, in 2022-11-23T03:54:46.1233403Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1233533Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1233724Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1233863Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1234065Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1234211Z self.run() 2022-11-23T03:54:46.1234402Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1234537Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1234884Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1235008Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1235371Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1235474Z getattr(self, test_name)() 2022-11-23T03:54:46.1235837Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1235924Z fn() 2022-11-23T03:54:46.1236292Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1236459Z test(self, **param_kwargs) 2022-11-23T03:54:46.1236824Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1236940Z return func(*args, **kwargs) 2022-11-23T03:54:46.1237170Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1237272Z self.run_subtests( 2022-11-23T03:54:46.1237625Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1237776Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1238144Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1238287Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1238670Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1238781Z output = model(*input) 2022-11-23T03:54:46.1239110Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1239242Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1239624Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1239777Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1240153Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1240265Z _lazy_init(state, module) 2022-11-23T03:54:46.1240617Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1240747Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1241093Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1241211Z return func(*args, **kwargs) 2022-11-23T03:54:46.1241591Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1241685Z p_assert( 2022-11-23T03:54:46.1242021Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1242135Z traceback.print_stack() 2022-11-23T03:54:46.1242359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1 2022-11-23T03:54:46.1242752Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 2 nodes. 2022-11-23T03:54:46.1242869Z File "", line 1, in 2022-11-23T03:54:46.1243066Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1243273Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1243472Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1243613Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1243801Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1243895Z self.run() 2022-11-23T03:54:46.1244084Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1244218Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1244562Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1244688Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1245052Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1245166Z getattr(self, test_name)() 2022-11-23T03:54:46.1245577Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1245673Z fn() 2022-11-23T03:54:46.1246041Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1246156Z test(self, **param_kwargs) 2022-11-23T03:54:46.1246517Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1246632Z return func(*args, **kwargs) 2022-11-23T03:54:46.1246861Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1246964Z self.run_subtests( 2022-11-23T03:54:46.1247319Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1247460Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1247968Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1248207Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1249035Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1249155Z output = model(*input) 2022-11-23T03:54:46.1249489Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1249622Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1250003Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1250168Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1250540Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1250650Z _lazy_init(state, module) 2022-11-23T03:54:46.1251012Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1251149Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1251491Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1251610Z return func(*args, **kwargs) 2022-11-23T03:54:46.1251996Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1252089Z p_assert( 2022-11-23T03:54:46.1252424Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1252540Z traceback.print_stack() 2022-11-23T03:54:46.1252751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0 2022-11-23T03:54:46.1253146Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 2 nodes. 2022-11-23T03:54:46.1253357Z File "", line 1, in 2022-11-23T03:54:46.1253555Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1253687Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1253875Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1254017Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1254222Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1254317Z self.run() 2022-11-23T03:54:46.1254511Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1254645Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1254994Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1255120Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1255546Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1255666Z getattr(self, test_name)() 2022-11-23T03:54:46.1256033Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1256125Z fn() 2022-11-23T03:54:46.1256480Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1256595Z test(self, **param_kwargs) 2022-11-23T03:54:46.1256955Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1257070Z return func(*args, **kwargs) 2022-11-23T03:54:46.1257300Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1257403Z self.run_subtests( 2022-11-23T03:54:46.1257768Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1257920Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1258288Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1258431Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1258810Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1258920Z output = model(*input) 2022-11-23T03:54:46.1259251Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1259382Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1259766Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1259939Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1260310Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1260426Z _lazy_init(state, module) 2022-11-23T03:54:46.1260766Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1260899Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1261243Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1261361Z return func(*args, **kwargs) 2022-11-23T03:54:46.1261746Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1261840Z p_assert( 2022-11-23T03:54:46.1262177Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1262351Z traceback.print_stack() 2022-11-23T03:54:46.1262579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1 2022-11-23T03:54:46.1262699Z File "", line 1, in 2022-11-23T03:54:46.1262895Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1263026Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1263219Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1263361Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1263562Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1263656Z self.run() 2022-11-23T03:54:46.1263852Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1263973Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1264367Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1264503Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1264875Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1264990Z getattr(self, test_name)() 2022-11-23T03:54:46.1265352Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1265441Z fn() 2022-11-23T03:54:46.1265812Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1265930Z test(self, **param_kwargs) 2022-11-23T03:54:46.1266290Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1266405Z return func(*args, **kwargs) 2022-11-23T03:54:46.1266642Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1266750Z self.run_subtests( 2022-11-23T03:54:46.1267106Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1267257Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1267626Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1267771Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1268156Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1268253Z output = model(*input) 2022-11-23T03:54:46.1268585Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1268720Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1269107Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1269274Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1269649Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1269767Z _lazy_init(state, module) 2022-11-23T03:54:46.1270124Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1270256Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1270598Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1270713Z return func(*args, **kwargs) 2022-11-23T03:54:46.1271097Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1271189Z p_assert( 2022-11-23T03:54:46.1271587Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1271700Z traceback.print_stack() 2022-11-23T03:54:46.1271924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0 2022-11-23T03:54:46.1272318Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 2 nodes. 2022-11-23T03:54:46.1272706Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 2 nodes. 2022-11-23T03:54:46.1272827Z File "", line 1, in 2022-11-23T03:54:46.1273011Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1273144Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1273334Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1273474Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1273723Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1273826Z self.run() 2022-11-23T03:54:46.1274021Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1274156Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1274500Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1274624Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1274995Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1275109Z getattr(self, test_name)() 2022-11-23T03:54:46.1275471Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1275559Z fn() 2022-11-23T03:54:46.1275933Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1276051Z test(self, **param_kwargs) 2022-11-23T03:54:46.1276416Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1276518Z return func(*args, **kwargs) 2022-11-23T03:54:46.1276749Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1276852Z self.run_subtests( 2022-11-23T03:54:46.1277207Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1277362Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1277730Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1277874Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1278262Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1278371Z output = model(*input) 2022-11-23T03:54:46.1278704Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1278835Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1279216Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1279381Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1279753Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1279865Z _lazy_init(state, module) 2022-11-23T03:54:46.1280222Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1280357Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1280757Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1280874Z return func(*args, **kwargs) 2022-11-23T03:54:46.1281243Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1281336Z p_assert( 2022-11-23T03:54:46.1281676Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1281791Z traceback.print_stack() 2022-11-23T03:54:46.1282013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1 2022-11-23T03:54:46.1282133Z File "", line 1, in 2022-11-23T03:54:46.1282329Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1282460Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1282704Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1282848Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1283050Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1283141Z self.run() 2022-11-23T03:54:46.1283332Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1283470Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1283818Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1283941Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1284295Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1284408Z getattr(self, test_name)() 2022-11-23T03:54:46.1284767Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1284868Z fn() 2022-11-23T03:54:46.1285238Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1285352Z test(self, **param_kwargs) 2022-11-23T03:54:46.1285718Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1285832Z return func(*args, **kwargs) 2022-11-23T03:54:46.1286061Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1286163Z self.run_subtests( 2022-11-23T03:54:46.1286517Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1286668Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1287035Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1287180Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1287566Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1287675Z output = model(*input) 2022-11-23T03:54:46.1288472Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1288660Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1289037Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1289203Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1289573Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1289686Z _lazy_init(state, module) 2022-11-23T03:54:46.1290046Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1290264Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1290611Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1290726Z return func(*args, **kwargs) 2022-11-23T03:54:46.1291103Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1291195Z p_assert( 2022-11-23T03:54:46.1291535Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1291646Z traceback.print_stack() 2022-11-23T03:54:46.1291867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0 2022-11-23T03:54:46.1292257Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 2 nodes. 2022-11-23T03:54:46.1292710Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 2 nodes. 2022-11-23T03:54:46.1292834Z File "", line 1, in 2022-11-23T03:54:46.1293030Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1293161Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1293352Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1293480Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1293677Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1293770Z self.run() 2022-11-23T03:54:46.1293963Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1294093Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1294444Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1294569Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1294934Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1295048Z getattr(self, test_name)() 2022-11-23T03:54:46.1295411Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1295497Z fn() 2022-11-23T03:54:46.1295863Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1295980Z test(self, **param_kwargs) 2022-11-23T03:54:46.1296341Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1296458Z return func(*args, **kwargs) 2022-11-23T03:54:46.1296685Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1296790Z self.run_subtests( 2022-11-23T03:54:46.1297136Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1297284Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1297649Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1297788Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1298164Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1298271Z output = model(*input) 2022-11-23T03:54:46.1298602Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1298735Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1299113Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1299332Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1299706Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1299814Z _lazy_init(state, module) 2022-11-23T03:54:46.1300166Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1300296Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1300632Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1300744Z return func(*args, **kwargs) 2022-11-23T03:54:46.1301125Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1301215Z p_assert( 2022-11-23T03:54:46.1301595Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1301704Z traceback.print_stack() 2022-11-23T03:54:46.1301929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 1 2022-11-23T03:54:46.1302319Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:8 with 2 nodes. 2022-11-23T03:54:46.1302434Z File "", line 1, in 2022-11-23T03:54:46.1302629Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1302757Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1302943Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1303081Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1303280Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1303371Z self.run() 2022-11-23T03:54:46.1303567Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1303698Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1304038Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1304157Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1304521Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1304634Z getattr(self, test_name)() 2022-11-23T03:54:46.1304996Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1305073Z fn() 2022-11-23T03:54:46.1305441Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1305552Z test(self, **param_kwargs) 2022-11-23T03:54:46.1305912Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1306026Z return func(*args, **kwargs) 2022-11-23T03:54:46.1306260Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1306358Z self.run_subtests( 2022-11-23T03:54:46.1306715Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1306864Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1307229Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1307368Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1307743Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1307851Z output = model(*input) 2022-11-23T03:54:46.1308247Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1308379Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1308758Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1308923Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1309293Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1309392Z _lazy_init(state, module) 2022-11-23T03:54:46.1309745Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1309878Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1310217Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1310396Z return func(*args, **kwargs) 2022-11-23T03:54:46.1310786Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1310875Z p_assert( 2022-11-23T03:54:46.1311214Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1311329Z traceback.print_stack() 2022-11-23T03:54:46.1311551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 0 2022-11-23T03:54:46.1311945Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:8 with 2 nodes. 2022-11-23T03:54:46.1312066Z File "", line 1, in 2022-11-23T03:54:46.1312261Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1312391Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1312582Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1312726Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1312927Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1313022Z self.run() 2022-11-23T03:54:46.1313201Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1313339Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1313681Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1313806Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1314175Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1314290Z getattr(self, test_name)() 2022-11-23T03:54:46.1314652Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1314744Z fn() 2022-11-23T03:54:46.1315115Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1315231Z test(self, **param_kwargs) 2022-11-23T03:54:46.1315594Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1315710Z return func(*args, **kwargs) 2022-11-23T03:54:46.1315940Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1316043Z self.run_subtests( 2022-11-23T03:54:46.1316395Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1316545Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1316912Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1317103Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1317484Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1317594Z output = model(*input) 2022-11-23T03:54:46.1317927Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1318060Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1318441Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1318607Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1318980Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1319093Z _lazy_init(state, module) 2022-11-23T03:54:46.1319496Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1319630Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1319972Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1320084Z return func(*args, **kwargs) 2022-11-23T03:54:46.1320465Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1320554Z p_assert( 2022-11-23T03:54:46.1320887Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1320998Z traceback.print_stack() 2022-11-23T03:54:46.1321218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 1 2022-11-23T03:54:46.1321325Z File "", line 1, in 2022-11-23T03:54:46.1321519Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1321659Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1321848Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1321985Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1322186Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1322281Z self.run() 2022-11-23T03:54:46.1322471Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1322605Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1322954Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1323076Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1323443Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1323561Z getattr(self, test_name)() 2022-11-23T03:54:46.1323927Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1324021Z fn() 2022-11-23T03:54:46.1324392Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1324505Z test(self, **param_kwargs) 2022-11-23T03:54:46.1324854Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1324970Z return func(*args, **kwargs) 2022-11-23T03:54:46.1325199Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1325302Z self.run_subtests( 2022-11-23T03:54:46.1325658Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1325810Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1326184Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1326381Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1326766Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1326875Z output = model(*input) 2022-11-23T03:54:46.1327205Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1327336Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1327811Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1328017Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1328403Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1328515Z _lazy_init(state, module) 2022-11-23T03:54:46.1328941Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1329076Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1329425Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1329528Z return func(*args, **kwargs) 2022-11-23T03:54:46.1329914Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1330010Z p_assert( 2022-11-23T03:54:46.1330351Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1330468Z traceback.print_stack() 2022-11-23T03:54:46.1330690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 0 2022-11-23T03:54:46.1331086Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:9 with 2 nodes. 2022-11-23T03:54:46.1331482Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:9 with 2 nodes. 2022-11-23T03:54:46.1331601Z File "", line 1, in 2022-11-23T03:54:46.1331798Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1331928Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1332118Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1332258Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1332462Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1332557Z self.run() 2022-11-23T03:54:46.1332751Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1332885Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1333222Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1333346Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1333717Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1333832Z getattr(self, test_name)() 2022-11-23T03:54:46.1334197Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1334289Z fn() 2022-11-23T03:54:46.1334657Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1334771Z test(self, **param_kwargs) 2022-11-23T03:54:46.1335132Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1335247Z return func(*args, **kwargs) 2022-11-23T03:54:46.1335480Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1335759Z self.run_subtests( 2022-11-23T03:54:46.1336123Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1336277Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1336647Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1336789Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1337171Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1337280Z output = model(*input) 2022-11-23T03:54:46.1337613Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1337731Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1338168Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1338335Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1338711Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1338825Z _lazy_init(state, module) 2022-11-23T03:54:46.1339182Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1339315Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1339657Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1339772Z return func(*args, **kwargs) 2022-11-23T03:54:46.1340157Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1340260Z p_assert( 2022-11-23T03:54:46.1340602Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1340717Z traceback.print_stack() 2022-11-23T03:54:46.1340941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 1 2022-11-23T03:54:46.1341060Z File "", line 1, in 2022-11-23T03:54:46.1341257Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1341390Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1341568Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1341713Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1341915Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1342010Z self.run() 2022-11-23T03:54:46.1342208Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1342346Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1342691Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1342814Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1343180Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1343295Z getattr(self, test_name)() 2022-11-23T03:54:46.1343658Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1343747Z fn() 2022-11-23T03:54:46.1344118Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1344234Z test(self, **param_kwargs) 2022-11-23T03:54:46.1344595Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1344769Z return func(*args, **kwargs) 2022-11-23T03:54:46.1345004Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1345097Z self.run_subtests( 2022-11-23T03:54:46.1345459Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1345615Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1345982Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1346125Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1346505Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1346615Z output = model(*input) 2022-11-23T03:54:46.1347020Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1347158Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1347545Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1347713Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1348087Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1348199Z _lazy_init(state, module) 2022-11-23T03:54:46.1348561Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1348693Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1349037Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1349152Z return func(*args, **kwargs) 2022-11-23T03:54:46.1349542Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1349638Z p_assert( 2022-11-23T03:54:46.1349963Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1350077Z traceback.print_stack() 2022-11-23T03:54:46.1350302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 0 2022-11-23T03:54:46.1350700Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:10 with 2 nodes. 2022-11-23T03:54:46.1351093Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:10 with 2 nodes. 2022-11-23T03:54:46.1351215Z File "", line 1, in 2022-11-23T03:54:46.1351416Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1351555Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1351750Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1351892Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1352094Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1352188Z self.run() 2022-11-23T03:54:46.1352380Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1352515Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1352861Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1352985Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1353352Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1353468Z getattr(self, test_name)() 2022-11-23T03:54:46.1353821Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1353967Z fn() 2022-11-23T03:54:46.1354341Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1354459Z test(self, **param_kwargs) 2022-11-23T03:54:46.1354823Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1354938Z return func(*args, **kwargs) 2022-11-23T03:54:46.1355167Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1355271Z self.run_subtests( 2022-11-23T03:54:46.1355627Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1355779Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1356193Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1356342Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1356721Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1356830Z output = model(*input) 2022-11-23T03:54:46.1357163Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1357299Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1357682Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1357849Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1358210Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1358324Z _lazy_init(state, module) 2022-11-23T03:54:46.1358686Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1358818Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1359165Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1359280Z return func(*args, **kwargs) 2022-11-23T03:54:46.1359662Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1359753Z p_assert( 2022-11-23T03:54:46.1360092Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1360213Z traceback.print_stack() 2022-11-23T03:54:46.1360443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 1 2022-11-23T03:54:46.1360562Z File "", line 1, in 2022-11-23T03:54:46.1360762Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1360898Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1361086Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1361227Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1361432Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1361514Z self.run() 2022-11-23T03:54:46.1361705Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1361842Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1362185Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1362309Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1362677Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1362851Z getattr(self, test_name)() 2022-11-23T03:54:46.1363219Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1363307Z fn() 2022-11-23T03:54:46.1363679Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1363795Z test(self, **param_kwargs) 2022-11-23T03:54:46.1364162Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1364277Z return func(*args, **kwargs) 2022-11-23T03:54:46.1364511Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1364614Z self.run_subtests( 2022-11-23T03:54:46.1364971Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1365169Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1365548Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1365681Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1366063Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1366172Z output = model(*input) 2022-11-23T03:54:46.1366512Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1366650Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1367032Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1367199Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1367576Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1367762Z _lazy_init(state, module) 2022-11-23T03:54:46.1368118Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1368251Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1368594Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1368716Z return func(*args, **kwargs) 2022-11-23T03:54:46.1369105Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1369198Z p_assert( 2022-11-23T03:54:46.1369536Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1369651Z traceback.print_stack() 2022-11-23T03:54:46.1369878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 0 2022-11-23T03:54:46.1370267Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:11 with 2 nodes. 2022-11-23T03:54:46.1370658Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:11 with 2 nodes. 2022-11-23T03:54:46.1370779Z File "", line 1, in 2022-11-23T03:54:46.1370979Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1371111Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1371303Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1371443Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1371644Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1371740Z self.run() 2022-11-23T03:54:46.1371933Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1372136Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1372485Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1372607Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1372975Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1373090Z getattr(self, test_name)() 2022-11-23T03:54:46.1373452Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1373544Z fn() 2022-11-23T03:54:46.1373916Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1374018Z test(self, **param_kwargs) 2022-11-23T03:54:46.1374380Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1374554Z return func(*args, **kwargs) 2022-11-23T03:54:46.1374793Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1374897Z self.run_subtests( 2022-11-23T03:54:46.1375257Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1375409Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1375779Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1375925Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1376306Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1376416Z output = model(*input) 2022-11-23T03:54:46.1376746Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1376887Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1377266Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1377434Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1377807Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1377921Z _lazy_init(state, module) 2022-11-23T03:54:46.1378275Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1378395Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1378737Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1378855Z return func(*args, **kwargs) 2022-11-23T03:54:46.1379240Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1379336Z p_assert( 2022-11-23T03:54:46.1379675Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1379790Z traceback.print_stack() 2022-11-23T03:54:46.1380015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 1 2022-11-23T03:54:46.1380138Z File "", line 1, in 2022-11-23T03:54:46.1380335Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1380467Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1380658Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1380798Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1381000Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1381169Z self.run() 2022-11-23T03:54:46.1381365Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1381504Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1381839Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1381964Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1382333Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1382447Z getattr(self, test_name)() 2022-11-23T03:54:46.1382809Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1382899Z fn() 2022-11-23T03:54:46.1383267Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1383383Z test(self, **param_kwargs) 2022-11-23T03:54:46.1383795Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1383917Z return func(*args, **kwargs) 2022-11-23T03:54:46.1384146Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1384249Z self.run_subtests( 2022-11-23T03:54:46.1384606Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1384760Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1385128Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1385272Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1385654Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1385767Z output = model(*input) 2022-11-23T03:54:46.1386093Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1386225Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1386606Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1386771Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1387144Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1387261Z _lazy_init(state, module) 2022-11-23T03:54:46.1387615Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1387748Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1388091Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1388212Z return func(*args, **kwargs) 2022-11-23T03:54:46.1388595Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1388687Z p_assert( 2022-11-23T03:54:46.1389028Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1389143Z traceback.print_stack() 2022-11-23T03:54:46.1389367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 0 2022-11-23T03:54:46.1389761Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:12 with 2 nodes. 2022-11-23T03:54:46.1390157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:12 with 2 nodes. 2022-11-23T03:54:46.1390276Z File "", line 1, in 2022-11-23T03:54:46.1390478Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1390652Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1390844Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1390986Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1391193Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1391288Z self.run() 2022-11-23T03:54:46.1391484Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1391621Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1391967Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1392092Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1392461Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1392576Z getattr(self, test_name)() 2022-11-23T03:54:46.1392988Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1393083Z fn() 2022-11-23T03:54:46.1393455Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1393570Z test(self, **param_kwargs) 2022-11-23T03:54:46.1393932Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1394048Z return func(*args, **kwargs) 2022-11-23T03:54:46.1394265Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1394370Z self.run_subtests( 2022-11-23T03:54:46.1394727Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1394878Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1395254Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1395398Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1395774Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1395884Z output = model(*input) 2022-11-23T03:54:46.1396214Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1396347Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1396730Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1396894Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1397266Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1397387Z _lazy_init(state, module) 2022-11-23T03:54:46.1397746Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1397879Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1398225Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1398340Z return func(*args, **kwargs) 2022-11-23T03:54:46.1398709Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1398805Z p_assert( 2022-11-23T03:54:46.1399143Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1399259Z traceback.print_stack() 2022-11-23T03:54:46.1399486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 1 2022-11-23T03:54:46.1399666Z File "", line 1, in 2022-11-23T03:54:46.1399867Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1399999Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1400188Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1400330Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1400532Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1400627Z self.run() 2022-11-23T03:54:46.1400820Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1400955Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1401300Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1401427Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1401839Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1401948Z getattr(self, test_name)() 2022-11-23T03:54:46.1402315Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1402404Z fn() 2022-11-23T03:54:46.1402776Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1402891Z test(self, **param_kwargs) 2022-11-23T03:54:46.1403253Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1403369Z return func(*args, **kwargs) 2022-11-23T03:54:46.1403601Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1403705Z self.run_subtests( 2022-11-23T03:54:46.1404066Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1404219Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1404587Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1404730Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1405114Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1405225Z output = model(*input) 2022-11-23T03:54:46.1405557Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1405689Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1406069Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1406221Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1406600Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1406712Z _lazy_init(state, module) 2022-11-23T03:54:46.1407067Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1407200Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1407542Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1407657Z return func(*args, **kwargs) 2022-11-23T03:54:46.1408497Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1408659Z p_assert( 2022-11-23T03:54:46.1409089Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1409204Z traceback.print_stack() 2022-11-23T03:54:46.1409522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 0 2022-11-23T03:54:46.1409923Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:13 with 2 nodes. 2022-11-23T03:54:46.1410315Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:13 with 2 nodes. 2022-11-23T03:54:46.1410435Z File "", line 1, in 2022-11-23T03:54:46.1410633Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1410765Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1410958Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1411101Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1411292Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1411387Z self.run() 2022-11-23T03:54:46.1411638Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1411776Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1412125Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1412253Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1412620Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1412733Z getattr(self, test_name)() 2022-11-23T03:54:46.1413098Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1413189Z fn() 2022-11-23T03:54:46.1413563Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1413677Z test(self, **param_kwargs) 2022-11-23T03:54:46.1414043Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1414165Z return func(*args, **kwargs) 2022-11-23T03:54:46.1414395Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1414495Z self.run_subtests( 2022-11-23T03:54:46.1414851Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1414988Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1415355Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1415499Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1415877Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1415986Z output = model(*input) 2022-11-23T03:54:46.1416322Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1416453Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1416834Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1416998Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1417372Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1417484Z _lazy_init(state, module) 2022-11-23T03:54:46.1417838Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1417969Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1418317Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1418434Z return func(*args, **kwargs) 2022-11-23T03:54:46.1418877Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1418970Z p_assert( 2022-11-23T03:54:46.1419310Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1419413Z traceback.print_stack() 2022-11-23T03:54:46.1419640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 1 2022-11-23T03:54:46.1419760Z File "", line 1, in 2022-11-23T03:54:46.1419957Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1420089Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1420281Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1420423Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1420671Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1420773Z self.run() 2022-11-23T03:54:46.1420967Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1421102Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1421453Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1421577Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1421947Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1422062Z getattr(self, test_name)() 2022-11-23T03:54:46.1422426Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1422515Z fn() 2022-11-23T03:54:46.1422870Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1422994Z test(self, **param_kwargs) 2022-11-23T03:54:46.1423356Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1423471Z return func(*args, **kwargs) 2022-11-23T03:54:46.1423703Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1423810Z self.run_subtests( 2022-11-23T03:54:46.1424169Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1424320Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1424688Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1424832Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1425223Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1425336Z output = model(*input) 2022-11-23T03:54:46.1425671Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1425804Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1426185Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1426352Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1426727Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1426842Z _lazy_init(state, module) 2022-11-23T03:54:46.1427185Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1427319Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1427665Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1427839Z return func(*args, **kwargs) 2022-11-23T03:54:46.1428228Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1428322Z p_assert( 2022-11-23T03:54:46.1428658Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1428773Z traceback.print_stack() 2022-11-23T03:54:46.1429002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 0 2022-11-23T03:54:46.1429398Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:14 with 2 nodes. 2022-11-23T03:54:46.1429790Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:14 with 2 nodes. 2022-11-23T03:54:46.1430058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 1 2022-11-23T03:54:46.1430289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 0 2022-11-23T03:54:46.1430683Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:15 with 2 nodes. 2022-11-23T03:54:46.1431073Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:15 with 2 nodes. 2022-11-23T03:54:46.1431295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 1 2022-11-23T03:54:46.1431517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 0 2022-11-23T03:54:46.1431914Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:16 with 2 nodes. 2022-11-23T03:54:46.1432312Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:16 with 2 nodes. 2022-11-23T03:54:46.1433093Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1433317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 1 2022-11-23T03:54:46.1433541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 0 2022-11-23T03:54:46.1433938Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:17 with 2 nodes. 2022-11-23T03:54:46.1434328Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:17 with 2 nodes. 2022-11-23T03:54:46.1434558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 0 2022-11-23T03:54:46.1434787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 1 2022-11-23T03:54:46.1435166Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:18 with 2 nodes. 2022-11-23T03:54:46.1435555Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:18 with 2 nodes. 2022-11-23T03:54:46.1436331Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1437119Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1437395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 0 2022-11-23T03:54:46.1437604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 1 2022-11-23T03:54:46.1437997Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:19 with 2 nodes. 2022-11-23T03:54:46.1438391Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:19 with 2 nodes. 2022-11-23T03:54:46.1439197Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1439431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:20 to store for rank: 1 2022-11-23T03:54:46.1439651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:20 to store for rank: 0 2022-11-23T03:54:46.1440049Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:20 with 2 nodes. 2022-11-23T03:54:46.1440438Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:20 with 2 nodes. 2022-11-23T03:54:46.1441205Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1441433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:21 to store for rank: 1 2022-11-23T03:54:46.1441662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:21 to store for rank: 0 2022-11-23T03:54:46.1442060Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:21 with 2 nodes. 2022-11-23T03:54:46.1442450Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:21 with 2 nodes. 2022-11-23T03:54:46.1442672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:22 to store for rank: 1 2022-11-23T03:54:46.1442892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:22 to store for rank: 0 2022-11-23T03:54:46.1443282Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:22 with 2 nodes. 2022-11-23T03:54:46.1443681Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:22 with 2 nodes. 2022-11-23T03:54:46.1444443Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1444668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:23 to store for rank: 0 2022-11-23T03:54:46.1444888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:23 to store for rank: 1 2022-11-23T03:54:46.1445278Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:23 with 2 nodes. 2022-11-23T03:54:46.1445666Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:23 with 2 nodes. 2022-11-23T03:54:46.1445890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:24 to store for rank: 0 2022-11-23T03:54:46.1446163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:24 to store for rank: 1 2022-11-23T03:54:46.1446554Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:24 with 2 nodes. 2022-11-23T03:54:46.1446944Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:24 with 2 nodes. 2022-11-23T03:54:46.1447766Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1447993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:25 to store for rank: 1 2022-11-23T03:54:46.1448282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:25 to store for rank: 0 2022-11-23T03:54:46.1448686Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:25 with 2 nodes. 2022-11-23T03:54:46.1449075Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:25 with 2 nodes. 2022-11-23T03:54:46.1449297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:26 to store for rank: 0 2022-11-23T03:54:46.1449517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:26 to store for rank: 1 2022-11-23T03:54:46.1449909Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:26 with 2 nodes. 2022-11-23T03:54:46.1450348Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:26 with 2 nodes. 2022-11-23T03:54:46.1451294Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1451567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:27 to store for rank: 1 2022-11-23T03:54:46.1451787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:27 to store for rank: 0 2022-11-23T03:54:46.1452188Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:27 with 2 nodes. 2022-11-23T03:54:46.1452576Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:27 with 2 nodes. 2022-11-23T03:54:46.1452796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:28 to store for rank: 1 2022-11-23T03:54:46.1453188Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:28 with 2 nodes. 2022-11-23T03:54:46.1453398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:28 to store for rank: 0 2022-11-23T03:54:46.1453787Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:28 with 2 nodes. 2022-11-23T03:54:46.1454545Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1454768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:29 to store for rank: 0 2022-11-23T03:54:46.1454987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:29 to store for rank: 1 2022-11-23T03:54:46.1455381Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:29 with 2 nodes. 2022-11-23T03:54:46.1455852Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:29 with 2 nodes. 2022-11-23T03:54:46.1456077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:30 to store for rank: 0 2022-11-23T03:54:46.1456298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:30 to store for rank: 1 2022-11-23T03:54:46.1456687Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:30 with 2 nodes. 2022-11-23T03:54:46.1457081Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:30 with 2 nodes. 2022-11-23T03:54:46.1457897Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1458128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:31 to store for rank: 1 2022-11-23T03:54:46.1458522Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:31 with 2 nodes. 2022-11-23T03:54:46.1458747Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:31 to store for rank: 0 2022-11-23T03:54:46.1459141Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:31 with 2 nodes. 2022-11-23T03:54:46.1459367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:32 to store for rank: 1 2022-11-23T03:54:46.1459586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:32 to store for rank: 0 2022-11-23T03:54:46.1459979Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:32 with 2 nodes. 2022-11-23T03:54:46.1460372Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:32 with 2 nodes. 2022-11-23T03:54:46.1461132Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1461354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:33 to store for rank: 1 2022-11-23T03:54:46.1461573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:33 to store for rank: 0 2022-11-23T03:54:46.1461961Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:33 with 2 nodes. 2022-11-23T03:54:46.1462356Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:33 with 2 nodes. 2022-11-23T03:54:46.1462575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:34 to store for rank: 1 2022-11-23T03:54:46.1462794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:34 to store for rank: 0 2022-11-23T03:54:46.1463184Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:34 with 2 nodes. 2022-11-23T03:54:46.1463573Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:34 with 2 nodes. 2022-11-23T03:54:46.1464332Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1465149Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1465372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:35 to store for rank: 1 2022-11-23T03:54:46.1465591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:35 to store for rank: 0 2022-11-23T03:54:46.1465979Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:35 with 2 nodes. 2022-11-23T03:54:46.1466367Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:35 with 2 nodes. 2022-11-23T03:54:46.1467163Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1467393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:36 to store for rank: 1 2022-11-23T03:54:46.1467786Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:36 with 2 nodes. 2022-11-23T03:54:46.1468007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:36 to store for rank: 0 2022-11-23T03:54:46.1468395Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:36 with 2 nodes. 2022-11-23T03:54:46.1469160Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1469385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:37 to store for rank: 1 2022-11-23T03:54:46.1469603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:37 to store for rank: 0 2022-11-23T03:54:46.1469992Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:37 with 2 nodes. 2022-11-23T03:54:46.1470382Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:37 with 2 nodes. 2022-11-23T03:54:46.1470487Z dist init r=1, world=2 2022-11-23T03:54:46.1470784Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.1471097Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.1471406Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.1471707Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.1472010Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.1472309Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.1472613Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.1472960Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.1473258Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.1473555Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.1473853Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.1474151Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.1474293Z dist init r=0, world=2 2022-11-23T03:54:46.1474607Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.1474907Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.1475205Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.1475502Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.1475799Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.1476107Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.1476404Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.1476701Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.1476999Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.1477297Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.1477597Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.1477896Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.1477989Z ok (9.242s) 2022-11-23T03:54:46.1478328Z test_mixture_of_experts_offload_true_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45411 2022-11-23T03:54:46.1478538Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45412 2022-11-23T03:54:46.1478922Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.1479089Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.1479474Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.1479703Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.1479928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.1480305Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.1480470Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.1480856Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.1481039Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.1481266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.1481646Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.1482081Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.1482363Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.1482637Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.1482856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.1483071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.1484128Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.1484233Z warnings.warn( 2022-11-23T03:54:46.1484458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T03:54:46.1485495Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.1485598Z warnings.warn( 2022-11-23T03:54:46.1485823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T03:54:46.1486213Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:54:46.1486612Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:54:46.1486733Z File "", line 1, in 2022-11-23T03:54:46.1486935Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1487070Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1487260Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1487403Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1487607Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1487757Z self.run() 2022-11-23T03:54:46.1487953Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1488090Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1488440Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1488629Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1488990Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1489107Z getattr(self, test_name)() 2022-11-23T03:54:46.1489472Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1489562Z fn() 2022-11-23T03:54:46.1489939Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1490054Z test(self, **param_kwargs) 2022-11-23T03:54:46.1490420Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1490535Z return func(*args, **kwargs) 2022-11-23T03:54:46.1490821Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1490932Z self.run_subtests( 2022-11-23T03:54:46.1491296Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1491450Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1491820Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1491963Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1492343Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1492453Z output = model(*input) 2022-11-23T03:54:46.1492784Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1492917Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1493292Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1493463Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1493837Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1493950Z _lazy_init(state, module) 2022-11-23T03:54:46.1494309Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1494443Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1494785Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1494905Z return func(*args, **kwargs) 2022-11-23T03:54:46.1495292Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1495385Z p_assert( 2022-11-23T03:54:46.1495729Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1495845Z traceback.print_stack() 2022-11-23T03:54:46.1496067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T03:54:46.1496187Z File "", line 1, in 2022-11-23T03:54:46.1496386Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1496518Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1496708Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1496851Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1497042Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1497138Z self.run() 2022-11-23T03:54:46.1497328Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1497525Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1497875Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1498001Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1498373Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1498492Z getattr(self, test_name)() 2022-11-23T03:54:46.1498858Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1498951Z fn() 2022-11-23T03:54:46.1499323Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1499440Z test(self, **param_kwargs) 2022-11-23T03:54:46.1499803Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1499965Z return func(*args, **kwargs) 2022-11-23T03:54:46.1500201Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1500304Z self.run_subtests( 2022-11-23T03:54:46.1500668Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1500807Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1501176Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1501322Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1501707Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1501817Z output = model(*input) 2022-11-23T03:54:46.1502149Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1502289Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1502672Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1502836Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1503208Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1503325Z _lazy_init(state, module) 2022-11-23T03:54:46.1503684Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1503817Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1504161Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1504276Z return func(*args, **kwargs) 2022-11-23T03:54:46.1504664Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1504760Z p_assert( 2022-11-23T03:54:46.1505100Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1505216Z traceback.print_stack() 2022-11-23T03:54:46.1505425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T03:54:46.1505821Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T03:54:46.1506215Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T03:54:46.1506334Z File "", line 1, in 2022-11-23T03:54:46.1506534Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1506668Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1506861Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1507055Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1507258Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1507354Z self.run() 2022-11-23T03:54:46.1507547Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1507685Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1508033Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1508160Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1508531Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1508647Z getattr(self, test_name)() 2022-11-23T03:54:46.1509012Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1509135Z fn() 2022-11-23T03:54:46.1509521Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1509637Z test(self, **param_kwargs) 2022-11-23T03:54:46.1510004Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1510120Z return func(*args, **kwargs) 2022-11-23T03:54:46.1510353Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1510457Z self.run_subtests( 2022-11-23T03:54:46.1510820Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1510974Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1511344Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1511495Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1511880Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1511992Z output = model(*input) 2022-11-23T03:54:46.1512325Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1512459Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1512844Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1513010Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1513385Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1513501Z _lazy_init(state, module) 2022-11-23T03:54:46.1513847Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1513983Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1514328Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1514443Z return func(*args, **kwargs) 2022-11-23T03:54:46.1514827Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1514923Z p_assert( 2022-11-23T03:54:46.1515263Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1515381Z traceback.print_stack() 2022-11-23T03:54:46.1515609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T03:54:46.1515729Z File "", line 1, in 2022-11-23T03:54:46.1515930Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1516139Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1516328Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1516471Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1516672Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1516767Z self.run() 2022-11-23T03:54:46.1516946Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1517086Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1517438Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1517562Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1517933Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1518051Z getattr(self, test_name)() 2022-11-23T03:54:46.1518482Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1518574Z fn() 2022-11-23T03:54:46.1518952Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1519070Z test(self, **param_kwargs) 2022-11-23T03:54:46.1519432Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1519548Z return func(*args, **kwargs) 2022-11-23T03:54:46.1519781Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1519886Z self.run_subtests( 2022-11-23T03:54:46.1520244Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1520395Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1520771Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1520914Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1521297Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1521393Z output = model(*input) 2022-11-23T03:54:46.1521727Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1521860Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1522242Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1522406Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1522781Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1522900Z _lazy_init(state, module) 2022-11-23T03:54:46.1523258Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1523391Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1523735Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1523852Z return func(*args, **kwargs) 2022-11-23T03:54:46.1524239Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1524334Z p_assert( 2022-11-23T03:54:46.1524675Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1524792Z traceback.print_stack() 2022-11-23T03:54:46.1525021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T03:54:46.1525419Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T03:54:46.1525900Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T03:54:46.1526007Z File "", line 1, in 2022-11-23T03:54:46.1526213Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1526350Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1526541Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1526684Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1526887Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1526985Z self.run() 2022-11-23T03:54:46.1527177Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1527312Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1527840Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1527977Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1528357Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1528474Z getattr(self, test_name)() 2022-11-23T03:54:46.1528842Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1528932Z fn() 2022-11-23T03:54:46.1529302Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1529419Z test(self, **param_kwargs) 2022-11-23T03:54:46.1529768Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1529887Z return func(*args, **kwargs) 2022-11-23T03:54:46.1530132Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1530240Z self.run_subtests( 2022-11-23T03:54:46.1530601Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1530755Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1531126Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1531268Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1531651Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1531765Z output = model(*input) 2022-11-23T03:54:46.1532098Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1532237Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1532624Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1532787Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1533161Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1533275Z _lazy_init(state, module) 2022-11-23T03:54:46.1533633Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1533770Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1534116Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1534219Z return func(*args, **kwargs) 2022-11-23T03:54:46.1534608Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1534775Z p_assert( 2022-11-23T03:54:46.1535119Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1535237Z traceback.print_stack() 2022-11-23T03:54:46.1535464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0 2022-11-23T03:54:46.1535586Z File "", line 1, in 2022-11-23T03:54:46.1535788Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1535921Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1536114Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1536255Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1536452Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1536547Z self.run() 2022-11-23T03:54:46.1536796Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1536937Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1537287Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1537412Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1537769Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1537885Z getattr(self, test_name)() 2022-11-23T03:54:46.1538253Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1538347Z fn() 2022-11-23T03:54:46.1538720Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1538837Z test(self, **param_kwargs) 2022-11-23T03:54:46.1539204Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1539325Z return func(*args, **kwargs) 2022-11-23T03:54:46.1539557Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1539661Z self.run_subtests( 2022-11-23T03:54:46.1540020Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1540173Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1540541Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1540684Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1541065Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1541176Z output = model(*input) 2022-11-23T03:54:46.1541511Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1541651Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1542019Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1542189Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1542566Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1542678Z _lazy_init(state, module) 2022-11-23T03:54:46.1543035Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1543168Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1543509Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1543623Z return func(*args, **kwargs) 2022-11-23T03:54:46.1544074Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1544167Z p_assert( 2022-11-23T03:54:46.1544508Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1544624Z traceback.print_stack() 2022-11-23T03:54:46.1544851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1 2022-11-23T03:54:46.1545244Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 2 nodes. 2022-11-23T03:54:46.1545632Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 2 nodes. 2022-11-23T03:54:46.1545754Z File "", line 1, in 2022-11-23T03:54:46.1545957Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1546140Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1546321Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1546467Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1546671Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1546767Z self.run() 2022-11-23T03:54:46.1546959Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1547098Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1547450Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1547575Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1547948Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1548064Z getattr(self, test_name)() 2022-11-23T03:54:46.1548437Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1548529Z fn() 2022-11-23T03:54:46.1548903Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1549017Z test(self, **param_kwargs) 2022-11-23T03:54:46.1549381Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1549498Z return func(*args, **kwargs) 2022-11-23T03:54:46.1549730Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1549821Z self.run_subtests( 2022-11-23T03:54:46.1550181Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1550335Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1550708Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1550856Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1551239Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1551350Z output = model(*input) 2022-11-23T03:54:46.1551682Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1551816Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1552199Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1552363Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1552737Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1552850Z _lazy_init(state, module) 2022-11-23T03:54:46.1553272Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1553405Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1553746Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1553861Z return func(*args, **kwargs) 2022-11-23T03:54:46.1554248Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1554343Z p_assert( 2022-11-23T03:54:46.1554669Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1554785Z traceback.print_stack() 2022-11-23T03:54:46.1555011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0 2022-11-23T03:54:46.1555453Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 2 nodes. 2022-11-23T03:54:46.1555581Z File "", line 1, in 2022-11-23T03:54:46.1555783Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1555920Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1556111Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1556253Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1556458Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1556554Z self.run() 2022-11-23T03:54:46.1556753Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1556887Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1557234Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1557363Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1557740Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1557856Z getattr(self, test_name)() 2022-11-23T03:54:46.1558207Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1558296Z fn() 2022-11-23T03:54:46.1558669Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1558785Z test(self, **param_kwargs) 2022-11-23T03:54:46.1559156Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1559276Z return func(*args, **kwargs) 2022-11-23T03:54:46.1559509Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1559613Z self.run_subtests( 2022-11-23T03:54:46.1559978Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1560132Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1560502Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1560646Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1561027Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1561137Z output = model(*input) 2022-11-23T03:54:46.1561471Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1561603Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1561986Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1562214Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1562590Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1562690Z _lazy_init(state, module) 2022-11-23T03:54:46.1563047Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1563179Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1563527Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1563642Z return func(*args, **kwargs) 2022-11-23T03:54:46.1564027Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1564120Z p_assert( 2022-11-23T03:54:46.1564459Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1564624Z traceback.print_stack() 2022-11-23T03:54:46.1564853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1 2022-11-23T03:54:46.1565254Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 2 nodes. 2022-11-23T03:54:46.1565375Z File "", line 1, in 2022-11-23T03:54:46.1565579Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1565713Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1565904Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1566045Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1566246Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1566329Z self.run() 2022-11-23T03:54:46.1566522Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1566668Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1567015Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1567139Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1567508Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1567626Z getattr(self, test_name)() 2022-11-23T03:54:46.1568048Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1568139Z fn() 2022-11-23T03:54:46.1568511Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1568630Z test(self, **param_kwargs) 2022-11-23T03:54:46.1568992Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1569114Z return func(*args, **kwargs) 2022-11-23T03:54:46.1569348Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1569452Z self.run_subtests( 2022-11-23T03:54:46.1569807Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1569960Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1570332Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1570463Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1570849Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1570958Z output = model(*input) 2022-11-23T03:54:46.1571293Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1571499Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1571888Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1572055Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1572428Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1572543Z _lazy_init(state, module) 2022-11-23T03:54:46.1572901Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1573034Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1573376Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1573491Z return func(*args, **kwargs) 2022-11-23T03:54:46.1573927Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1574028Z p_assert( 2022-11-23T03:54:46.1574372Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1574487Z traceback.print_stack() 2022-11-23T03:54:46.1574712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0 2022-11-23T03:54:46.1574820Z File "", line 1, in 2022-11-23T03:54:46.1575019Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1575151Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1575345Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1575486Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1575692Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1575793Z self.run() 2022-11-23T03:54:46.1575990Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1576130Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1576474Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1576598Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1576970Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1577085Z getattr(self, test_name)() 2022-11-23T03:54:46.1577451Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1577541Z fn() 2022-11-23T03:54:46.1577912Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1578025Z test(self, **param_kwargs) 2022-11-23T03:54:46.1578383Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1578501Z return func(*args, **kwargs) 2022-11-23T03:54:46.1578740Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1578846Z self.run_subtests( 2022-11-23T03:54:46.1579203Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1579360Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1579729Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1579875Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1580257Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1580432Z output = model(*input) 2022-11-23T03:54:46.1580763Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1580896Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1581277Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1581444Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1581818Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1581932Z _lazy_init(state, module) 2022-11-23T03:54:46.1582288Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1582420Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1582750Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1582933Z return func(*args, **kwargs) 2022-11-23T03:54:46.1583327Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1583422Z p_assert( 2022-11-23T03:54:46.1583763Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1583883Z traceback.print_stack() 2022-11-23T03:54:46.1584107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1 2022-11-23T03:54:46.1584502Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 2 nodes. 2022-11-23T03:54:46.1584890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 2 nodes. 2022-11-23T03:54:46.1585006Z File "", line 1, in 2022-11-23T03:54:46.1585204Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1585341Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1585528Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1585665Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1585863Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1585953Z self.run() 2022-11-23T03:54:46.1586141Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1586273Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1586605Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1586731Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1587101Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1587225Z getattr(self, test_name)() 2022-11-23T03:54:46.1587589Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1587680Z fn() 2022-11-23T03:54:46.1588052Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1588173Z test(self, **param_kwargs) 2022-11-23T03:54:46.1588539Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1588657Z return func(*args, **kwargs) 2022-11-23T03:54:46.1588887Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1588990Z self.run_subtests( 2022-11-23T03:54:46.1589346Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1589502Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1589934Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1590079Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1590466Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1590578Z output = model(*input) 2022-11-23T03:54:46.1590898Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1591031Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1591414Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1591579Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1592001Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1592115Z _lazy_init(state, module) 2022-11-23T03:54:46.1592477Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1592611Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1592958Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1593073Z return func(*args, **kwargs) 2022-11-23T03:54:46.1593458Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1593552Z p_assert( 2022-11-23T03:54:46.1593892Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1594010Z traceback.print_stack() 2022-11-23T03:54:46.1594235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 0 2022-11-23T03:54:46.1594638Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:8 with 2 nodes. 2022-11-23T03:54:46.1594761Z File "", line 1, in 2022-11-23T03:54:46.1594961Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1595081Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1595270Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1595411Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1595614Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1595709Z self.run() 2022-11-23T03:54:46.1595900Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1596034Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1596383Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1596512Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1596881Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1596996Z getattr(self, test_name)() 2022-11-23T03:54:46.1597366Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1597456Z fn() 2022-11-23T03:54:46.1597828Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1597943Z test(self, **param_kwargs) 2022-11-23T03:54:46.1598308Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1598424Z return func(*args, **kwargs) 2022-11-23T03:54:46.1598643Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1598813Z self.run_subtests( 2022-11-23T03:54:46.1599178Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1599331Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1599702Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1599847Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1600232Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1600342Z output = model(*input) 2022-11-23T03:54:46.1600674Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1600808Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1601235Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1601409Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1601786Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1601899Z _lazy_init(state, module) 2022-11-23T03:54:46.1602258Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1602390Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1602735Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1602850Z return func(*args, **kwargs) 2022-11-23T03:54:46.1603235Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1603315Z p_assert( 2022-11-23T03:54:46.1603658Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1603779Z traceback.print_stack() 2022-11-23T03:54:46.1604005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 1 2022-11-23T03:54:46.1604399Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:8 with 2 nodes. 2022-11-23T03:54:46.1604519Z File "", line 1, in 2022-11-23T03:54:46.1604719Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1604853Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1605047Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1605190Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1605393Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1605494Z self.run() 2022-11-23T03:54:46.1605696Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1605831Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1606178Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1606302Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1606670Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1606771Z getattr(self, test_name)() 2022-11-23T03:54:46.1607139Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1607228Z fn() 2022-11-23T03:54:46.1607600Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1607766Z test(self, **param_kwargs) 2022-11-23T03:54:46.1608136Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1608316Z return func(*args, **kwargs) 2022-11-23T03:54:46.1608550Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1608655Z self.run_subtests( 2022-11-23T03:54:46.1609017Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1609173Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1609543Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1609687Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1610070Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1610179Z output = model(*input) 2022-11-23T03:54:46.1610562Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1610696Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1611085Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1611239Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1611616Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1611732Z _lazy_init(state, module) 2022-11-23T03:54:46.1612090Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1612221Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1612565Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1612687Z return func(*args, **kwargs) 2022-11-23T03:54:46.1613069Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1613167Z p_assert( 2022-11-23T03:54:46.1613507Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1613625Z traceback.print_stack() 2022-11-23T03:54:46.1613848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 0 2022-11-23T03:54:46.1613969Z File "", line 1, in 2022-11-23T03:54:46.1614169Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1614302Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1614494Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1614636Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1614841Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1614926Z self.run() 2022-11-23T03:54:46.1615119Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1615255Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1615602Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1615728Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1616098Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1616217Z getattr(self, test_name)() 2022-11-23T03:54:46.1616584Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1616673Z fn() 2022-11-23T03:54:46.1617045Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1617226Z test(self, **param_kwargs) 2022-11-23T03:54:46.1617593Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1617709Z return func(*args, **kwargs) 2022-11-23T03:54:46.1617940Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1618043Z self.run_subtests( 2022-11-23T03:54:46.1618402Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1618557Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1618913Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1619056Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1619596Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1619713Z output = model(*input) 2022-11-23T03:54:46.1620052Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1620184Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1620568Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1620735Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1621111Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1621220Z _lazy_init(state, module) 2022-11-23T03:54:46.1621579Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1621716Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1622066Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1622186Z return func(*args, **kwargs) 2022-11-23T03:54:46.1622571Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1622669Z p_assert( 2022-11-23T03:54:46.1623011Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1623131Z traceback.print_stack() 2022-11-23T03:54:46.1623341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 1 2022-11-23T03:54:46.1623735Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:9 with 2 nodes. 2022-11-23T03:54:46.1624128Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:9 with 2 nodes. 2022-11-23T03:54:46.1624252Z File "", line 1, in 2022-11-23T03:54:46.1624457Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1624593Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1624787Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1624929Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1625131Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1625226Z self.run() 2022-11-23T03:54:46.1625419Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1625554Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1625901Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1626030Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1626403Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1626578Z getattr(self, test_name)() 2022-11-23T03:54:46.1626948Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1627039Z fn() 2022-11-23T03:54:46.1627398Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1627515Z test(self, **param_kwargs) 2022-11-23T03:54:46.1627879Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1627996Z return func(*args, **kwargs) 2022-11-23T03:54:46.1628232Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1628336Z self.run_subtests( 2022-11-23T03:54:46.1628740Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1628898Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1629272Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1629419Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1629800Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1629910Z output = model(*input) 2022-11-23T03:54:46.1630243Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1630376Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1630762Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1630926Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1631309Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1631423Z _lazy_init(state, module) 2022-11-23T03:54:46.1631766Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1631900Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1632243Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1632359Z return func(*args, **kwargs) 2022-11-23T03:54:46.1632749Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1632843Z p_assert( 2022-11-23T03:54:46.1633183Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1633299Z traceback.print_stack() 2022-11-23T03:54:46.1633532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 0 2022-11-23T03:54:46.1633653Z File "", line 1, in 2022-11-23T03:54:46.1633850Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1633984Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1634175Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1634321Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1634523Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1634623Z self.run() 2022-11-23T03:54:46.1634815Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1634936Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1635278Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1635460Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1635835Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1635951Z getattr(self, test_name)() 2022-11-23T03:54:46.1636317Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1636407Z fn() 2022-11-23T03:54:46.1636779Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1636894Z test(self, **param_kwargs) 2022-11-23T03:54:46.1637259Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1637376Z return func(*args, **kwargs) 2022-11-23T03:54:46.1637609Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1637712Z self.run_subtests( 2022-11-23T03:54:46.1638122Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1638279Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1638652Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1638797Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1639177Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1639273Z output = model(*input) 2022-11-23T03:54:46.1639608Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1639741Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1640127Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1640303Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1640676Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1640788Z _lazy_init(state, module) 2022-11-23T03:54:46.1641148Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1641280Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1641624Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1641740Z return func(*args, **kwargs) 2022-11-23T03:54:46.1642125Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1642219Z p_assert( 2022-11-23T03:54:46.1642560Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1642686Z traceback.print_stack() 2022-11-23T03:54:46.1642911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 1 2022-11-23T03:54:46.1643305Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:10 with 2 nodes. 2022-11-23T03:54:46.1643701Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:10 with 2 nodes. 2022-11-23T03:54:46.1643822Z File "", line 1, in 2022-11-23T03:54:46.1644009Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1644143Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1644335Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1644477Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1644685Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1644842Z self.run() 2022-11-23T03:54:46.1645035Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1645172Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1645523Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1645646Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1646015Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1646130Z getattr(self, test_name)() 2022-11-23T03:54:46.1646495Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1646586Z fn() 2022-11-23T03:54:46.1646963Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1647129Z test(self, **param_kwargs) 2022-11-23T03:54:46.1647500Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1647603Z return func(*args, **kwargs) 2022-11-23T03:54:46.1647969Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1648075Z self.run_subtests( 2022-11-23T03:54:46.1648438Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1648590Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1648960Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1649104Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1649488Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1649602Z output = model(*input) 2022-11-23T03:54:46.1649937Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1650069Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1650451Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1650614Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1650989Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1651101Z _lazy_init(state, module) 2022-11-23T03:54:46.1651458Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1651590Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1651935Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1652056Z return func(*args, **kwargs) 2022-11-23T03:54:46.1652427Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1652521Z p_assert( 2022-11-23T03:54:46.1652860Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1652977Z traceback.print_stack() 2022-11-23T03:54:46.1653204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 0 2022-11-23T03:54:46.1653326Z File "", line 1, in 2022-11-23T03:54:46.1653523Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1653657Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1653849Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1654078Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1654280Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1654376Z self.run() 2022-11-23T03:54:46.1654568Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1654704Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1655052Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1655177Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1655533Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1655651Z getattr(self, test_name)() 2022-11-23T03:54:46.1656018Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1656108Z fn() 2022-11-23T03:54:46.1656534Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1656658Z test(self, **param_kwargs) 2022-11-23T03:54:46.1657028Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1657149Z return func(*args, **kwargs) 2022-11-23T03:54:46.1657380Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1657484Z self.run_subtests( 2022-11-23T03:54:46.1657841Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1657999Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1658369Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1658511Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1658901Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1659011Z output = model(*input) 2022-11-23T03:54:46.1659342Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1659474Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1659842Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1660009Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1660382Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1660496Z _lazy_init(state, module) 2022-11-23T03:54:46.1660854Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1660996Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1661339Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1661458Z return func(*args, **kwargs) 2022-11-23T03:54:46.1661843Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1661935Z p_assert( 2022-11-23T03:54:46.1662275Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1662391Z traceback.print_stack() 2022-11-23T03:54:46.1662615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 1 2022-11-23T03:54:46.1663012Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:11 with 2 nodes. 2022-11-23T03:54:46.1663408Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:11 with 2 nodes. 2022-11-23T03:54:46.1663587Z File "", line 1, in 2022-11-23T03:54:46.1663786Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1663919Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1664110Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1664238Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1664442Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1664538Z self.run() 2022-11-23T03:54:46.1664731Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1664867Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1665220Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1665344Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1665763Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1665881Z getattr(self, test_name)() 2022-11-23T03:54:46.1666248Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1666337Z fn() 2022-11-23T03:54:46.1666708Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1666823Z test(self, **param_kwargs) 2022-11-23T03:54:46.1667186Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1667302Z return func(*args, **kwargs) 2022-11-23T03:54:46.1667534Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1667637Z self.run_subtests( 2022-11-23T03:54:46.1667990Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1668143Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1668512Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1668657Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1669039Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1669151Z output = model(*input) 2022-11-23T03:54:46.1669485Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1669617Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1669999Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1670170Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1670543Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1670657Z _lazy_init(state, module) 2022-11-23T03:54:46.1671013Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1671147Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1671489Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1671605Z return func(*args, **kwargs) 2022-11-23T03:54:46.1671990Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1672084Z p_assert( 2022-11-23T03:54:46.1672427Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1672591Z traceback.print_stack() 2022-11-23T03:54:46.1672817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 1 2022-11-23T03:54:46.1673217Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:12 with 2 nodes. 2022-11-23T03:54:46.1673337Z File "", line 1, in 2022-11-23T03:54:46.1673537Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1673671Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1673862Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1674004Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1674205Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1674298Z self.run() 2022-11-23T03:54:46.1674486Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1674671Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1675029Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1675155Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1675523Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1675639Z getattr(self, test_name)() 2022-11-23T03:54:46.1676007Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1676083Z fn() 2022-11-23T03:54:46.1676453Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1676570Z test(self, **param_kwargs) 2022-11-23T03:54:46.1676933Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1677061Z return func(*args, **kwargs) 2022-11-23T03:54:46.1677291Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1677396Z self.run_subtests( 2022-11-23T03:54:46.1677755Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1677907Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1678276Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1678421Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1678802Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1678913Z output = model(*input) 2022-11-23T03:54:46.1679249Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1679385Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1679767Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1679933Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1680311Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1680411Z _lazy_init(state, module) 2022-11-23T03:54:46.1680767Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1680900Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1681243Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1681359Z return func(*args, **kwargs) 2022-11-23T03:54:46.1681747Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1681907Z p_assert( 2022-11-23T03:54:46.1682252Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1682368Z traceback.print_stack() 2022-11-23T03:54:46.1682593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 0 2022-11-23T03:54:46.1682991Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:12 with 2 nodes. 2022-11-23T03:54:46.1683110Z File "", line 1, in 2022-11-23T03:54:46.1683311Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1683449Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1683641Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1683783Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1684039Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1684140Z self.run() 2022-11-23T03:54:46.1684320Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1684456Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1684807Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1684931Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1685300Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1685415Z getattr(self, test_name)() 2022-11-23T03:54:46.1685782Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1685872Z fn() 2022-11-23T03:54:46.1686248Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1686365Z test(self, **param_kwargs) 2022-11-23T03:54:46.1686728Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1686845Z return func(*args, **kwargs) 2022-11-23T03:54:46.1687079Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1687183Z self.run_subtests( 2022-11-23T03:54:46.1687543Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1687762Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1688133Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1688264Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1688661Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1688773Z output = model(*input) 2022-11-23T03:54:46.1689104Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1689237Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1689621Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1689785Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1690159Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1690273Z _lazy_init(state, module) 2022-11-23T03:54:46.1690631Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1690764Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1691181Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1691300Z return func(*args, **kwargs) 2022-11-23T03:54:46.1691686Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1691780Z p_assert( 2022-11-23T03:54:46.1692123Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1692237Z traceback.print_stack() 2022-11-23T03:54:46.1692463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 1 2022-11-23T03:54:46.1692858Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:13 with 2 nodes. 2022-11-23T03:54:46.1692965Z File "", line 1, in 2022-11-23T03:54:46.1693214Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1693354Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1693549Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1693690Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1693895Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1693990Z self.run() 2022-11-23T03:54:46.1694187Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1694321Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1694671Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1694795Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1695165Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1695280Z getattr(self, test_name)() 2022-11-23T03:54:46.1695651Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1695742Z fn() 2022-11-23T03:54:46.1696114Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1696229Z test(self, **param_kwargs) 2022-11-23T03:54:46.1696582Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1696698Z return func(*args, **kwargs) 2022-11-23T03:54:46.1696930Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1697035Z self.run_subtests( 2022-11-23T03:54:46.1697393Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1697547Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1697920Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1698066Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1698449Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1698557Z output = model(*input) 2022-11-23T03:54:46.1698890Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1699021Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1699406Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1699571Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1699949Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1700126Z _lazy_init(state, module) 2022-11-23T03:54:46.1700486Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1700623Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1700955Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1701073Z return func(*args, **kwargs) 2022-11-23T03:54:46.1701460Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1701555Z p_assert( 2022-11-23T03:54:46.1701896Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1702012Z traceback.print_stack() 2022-11-23T03:54:46.1702244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 0 2022-11-23T03:54:46.1702713Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:13 with 2 nodes. 2022-11-23T03:54:46.1702836Z File "", line 1, in 2022-11-23T03:54:46.1703035Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1703168Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1703359Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1703501Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1703703Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1703797Z self.run() 2022-11-23T03:54:46.1703992Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1704125Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1704479Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1704594Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1704965Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1705082Z getattr(self, test_name)() 2022-11-23T03:54:46.1705449Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1705540Z fn() 2022-11-23T03:54:46.1705911Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1706027Z test(self, **param_kwargs) 2022-11-23T03:54:46.1706390Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1706506Z return func(*args, **kwargs) 2022-11-23T03:54:46.1706740Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1706850Z self.run_subtests( 2022-11-23T03:54:46.1707215Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1707370Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1707738Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1707882Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1708264Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1708374Z output = model(*input) 2022-11-23T03:54:46.1708709Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1708829Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1709213Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1709442Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1709819Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1709933Z _lazy_init(state, module) 2022-11-23T03:54:46.1710291Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1710423Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1710766Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1710882Z return func(*args, **kwargs) 2022-11-23T03:54:46.1711268Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1711363Z p_assert( 2022-11-23T03:54:46.1711745Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1711868Z traceback.print_stack() 2022-11-23T03:54:46.1712094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 0 2022-11-23T03:54:46.1712213Z File "", line 1, in 2022-11-23T03:54:46.1712411Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1712545Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1712725Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1712868Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1713073Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1713167Z self.run() 2022-11-23T03:54:46.1713359Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1713495Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1713852Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1713978Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1714351Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1714466Z getattr(self, test_name)() 2022-11-23T03:54:46.1714833Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1714922Z fn() 2022-11-23T03:54:46.1715298Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1715414Z test(self, **param_kwargs) 2022-11-23T03:54:46.1715777Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1715894Z return func(*args, **kwargs) 2022-11-23T03:54:46.1716130Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 240, in test_mixture_of_experts 2022-11-23T03:54:46.1716222Z self.run_subtests( 2022-11-23T03:54:46.1716585Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1716739Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1717109Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1717255Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1717639Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1717750Z output = model(*input) 2022-11-23T03:54:46.1718084Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1718222Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1718673Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1718839Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1719215Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1719329Z _lazy_init(state, module) 2022-11-23T03:54:46.1719689Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1719821Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1720169Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1720286Z return func(*args, **kwargs) 2022-11-23T03:54:46.1720730Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1720831Z p_assert( 2022-11-23T03:54:46.1721160Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1721277Z traceback.print_stack() 2022-11-23T03:54:46.1721506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 1 2022-11-23T03:54:46.1721901Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:14 with 2 nodes. 2022-11-23T03:54:46.1722295Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:14 with 2 nodes. 2022-11-23T03:54:46.1722519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 1 2022-11-23T03:54:46.1722911Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:15 with 2 nodes. 2022-11-23T03:54:46.1723138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 0 2022-11-23T03:54:46.1723535Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:15 with 2 nodes. 2022-11-23T03:54:46.1723760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 1 2022-11-23T03:54:46.1723982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 0 2022-11-23T03:54:46.1724375Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:16 with 2 nodes. 2022-11-23T03:54:46.1724765Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:16 with 2 nodes. 2022-11-23T03:54:46.1725540Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1725767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 1 2022-11-23T03:54:46.1725992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 0 2022-11-23T03:54:46.1726385Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:17 with 2 nodes. 2022-11-23T03:54:46.1726778Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:17 with 2 nodes. 2022-11-23T03:54:46.1727000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 1 2022-11-23T03:54:46.1727221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 0 2022-11-23T03:54:46.1727618Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:18 with 2 nodes. 2022-11-23T03:54:46.1728121Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:18 with 2 nodes. 2022-11-23T03:54:46.1728895Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1729120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 0 2022-11-23T03:54:46.1729339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 1 2022-11-23T03:54:46.1729728Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:19 with 2 nodes. 2022-11-23T03:54:46.1730175Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:19 with 2 nodes. 2022-11-23T03:54:46.1730937Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1731161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:20 to store for rank: 1 2022-11-23T03:54:46.1731381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:20 to store for rank: 0 2022-11-23T03:54:46.1731775Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:20 with 2 nodes. 2022-11-23T03:54:46.1732167Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:20 with 2 nodes. 2022-11-23T03:54:46.1732942Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1733164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:21 to store for rank: 1 2022-11-23T03:54:46.1733386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:21 to store for rank: 0 2022-11-23T03:54:46.1733777Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:21 with 2 nodes. 2022-11-23T03:54:46.1734169Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:21 with 2 nodes. 2022-11-23T03:54:46.1734378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:22 to store for rank: 1 2022-11-23T03:54:46.1734605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:22 to store for rank: 0 2022-11-23T03:54:46.1734995Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:22 with 2 nodes. 2022-11-23T03:54:46.1735386Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:22 with 2 nodes. 2022-11-23T03:54:46.1736146Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1736372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:23 to store for rank: 0 2022-11-23T03:54:46.1736598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:23 to store for rank: 1 2022-11-23T03:54:46.1737054Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:23 with 2 nodes. 2022-11-23T03:54:46.1737443Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:23 with 2 nodes. 2022-11-23T03:54:46.1737666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:24 to store for rank: 0 2022-11-23T03:54:46.1738060Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:24 with 2 nodes. 2022-11-23T03:54:46.1738281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:24 to store for rank: 1 2022-11-23T03:54:46.1738670Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:24 with 2 nodes. 2022-11-23T03:54:46.1739482Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1739713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:25 to store for rank: 1 2022-11-23T03:54:46.1739932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:25 to store for rank: 0 2022-11-23T03:54:46.1740323Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:25 with 2 nodes. 2022-11-23T03:54:46.1740713Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:25 with 2 nodes. 2022-11-23T03:54:46.1740939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:26 to store for rank: 0 2022-11-23T03:54:46.1741165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:26 to store for rank: 1 2022-11-23T03:54:46.1741557Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:26 with 2 nodes. 2022-11-23T03:54:46.1741952Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:26 with 2 nodes. 2022-11-23T03:54:46.1742724Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1742951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:27 to store for rank: 0 2022-11-23T03:54:46.1743342Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:27 with 2 nodes. 2022-11-23T03:54:46.1743570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:27 to store for rank: 1 2022-11-23T03:54:46.1743963Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:27 with 2 nodes. 2022-11-23T03:54:46.1744187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:28 to store for rank: 0 2022-11-23T03:54:46.1744403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:28 to store for rank: 1 2022-11-23T03:54:46.1744793Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:28 with 2 nodes. 2022-11-23T03:54:46.1745186Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:28 with 2 nodes. 2022-11-23T03:54:46.1745953Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1746240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:29 to store for rank: 1 2022-11-23T03:54:46.1746461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:29 to store for rank: 0 2022-11-23T03:54:46.1746856Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:29 with 2 nodes. 2022-11-23T03:54:46.1747245Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:29 with 2 nodes. 2022-11-23T03:54:46.1747468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:30 to store for rank: 0 2022-11-23T03:54:46.1747690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:30 to store for rank: 1 2022-11-23T03:54:46.1748111Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:30 with 2 nodes. 2022-11-23T03:54:46.1748511Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:30 with 2 nodes. 2022-11-23T03:54:46.1749277Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1749502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:31 to store for rank: 0 2022-11-23T03:54:46.1749723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:31 to store for rank: 1 2022-11-23T03:54:46.1750116Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:31 with 2 nodes. 2022-11-23T03:54:46.1750510Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:31 with 2 nodes. 2022-11-23T03:54:46.1750735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:32 to store for rank: 0 2022-11-23T03:54:46.1751129Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:32 with 2 nodes. 2022-11-23T03:54:46.1751349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:32 to store for rank: 1 2022-11-23T03:54:46.1751737Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:32 with 2 nodes. 2022-11-23T03:54:46.1752496Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1752727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:33 to store for rank: 1 2022-11-23T03:54:46.1752951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:33 to store for rank: 0 2022-11-23T03:54:46.1753342Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:33 with 2 nodes. 2022-11-23T03:54:46.1753739Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:33 with 2 nodes. 2022-11-23T03:54:46.1753964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:34 to store for rank: 1 2022-11-23T03:54:46.1754190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:34 to store for rank: 0 2022-11-23T03:54:46.1754582Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:34 with 2 nodes. 2022-11-23T03:54:46.1754975Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:34 with 2 nodes. 2022-11-23T03:54:46.1755789Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1756557Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1756783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:35 to store for rank: 1 2022-11-23T03:54:46.1757002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:35 to store for rank: 0 2022-11-23T03:54:46.1757436Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:35 with 2 nodes. 2022-11-23T03:54:46.1757832Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:35 with 2 nodes. 2022-11-23T03:54:46.1758589Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1758813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:36 to store for rank: 0 2022-11-23T03:54:46.1759034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:36 to store for rank: 1 2022-11-23T03:54:46.1759433Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:36 with 2 nodes. 2022-11-23T03:54:46.1759826Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:36 with 2 nodes. 2022-11-23T03:54:46.1760590Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1760815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:37 to store for rank: 0 2022-11-23T03:54:46.1761037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:37 to store for rank: 1 2022-11-23T03:54:46.1761428Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:37 with 2 nodes. 2022-11-23T03:54:46.1761826Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:37 with 2 nodes. 2022-11-23T03:54:46.1761932Z dist init r=1, world=2 2022-11-23T03:54:46.1762248Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.1762556Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.1762860Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.1763162Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.1763468Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.1763816Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.1764116Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.1764415Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.1764728Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.1765036Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.1765381Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.1765687Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.1765790Z dist init r=0, world=2 2022-11-23T03:54:46.1766075Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.1766375Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.1766673Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.1766977Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.1767278Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.1767576Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.1767985Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.1768288Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.1768590Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.1768894Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.1769190Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.1769488Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.1769583Z ok (9.140s) 2022-11-23T03:54:46.1769930Z test_mixture_of_experts_with_delay_before_free_offload_false_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45828 2022-11-23T03:54:46.1770146Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45829 2022-11-23T03:54:46.1770597Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.1770764Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.1771156Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.1771336Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.1771558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.1771934Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.1772098Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.1772532Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.1772717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.1772940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.1773341Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.1773733Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.1774011Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.1774287Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.1774504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.1774722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.1775790Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.1775893Z warnings.warn( 2022-11-23T03:54:46.1776118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T03:54:46.1777165Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.1777272Z warnings.warn( 2022-11-23T03:54:46.1777497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T03:54:46.1777876Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:54:46.1778268Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:54:46.1778491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T03:54:46.1778716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T03:54:46.1779106Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T03:54:46.1779554Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T03:54:46.1779774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T03:54:46.1779995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T03:54:46.1780385Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T03:54:46.1780774Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T03:54:46.1781539Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1782348Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1782575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0 2022-11-23T03:54:46.1782797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1 2022-11-23T03:54:46.1783193Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 2 nodes. 2022-11-23T03:54:46.1783580Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 2 nodes. 2022-11-23T03:54:46.1783803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0 2022-11-23T03:54:46.1784037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1 2022-11-23T03:54:46.1784426Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 2 nodes. 2022-11-23T03:54:46.1784812Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 2 nodes. 2022-11-23T03:54:46.1785037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0 2022-11-23T03:54:46.1785259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1 2022-11-23T03:54:46.1785652Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 2 nodes. 2022-11-23T03:54:46.1786044Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 2 nodes. 2022-11-23T03:54:46.1786818Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1787582Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1788340Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1789163Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1789920Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1790677Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1790946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 0 2022-11-23T03:54:46.1791171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 1 2022-11-23T03:54:46.1791568Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:8 with 2 nodes. 2022-11-23T03:54:46.1791960Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:8 with 2 nodes. 2022-11-23T03:54:46.1792182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 0 2022-11-23T03:54:46.1792402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 1 2022-11-23T03:54:46.1792791Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:9 with 2 nodes. 2022-11-23T03:54:46.1793186Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:9 with 2 nodes. 2022-11-23T03:54:46.1793412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 0 2022-11-23T03:54:46.1793633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 1 2022-11-23T03:54:46.1794030Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:10 with 2 nodes. 2022-11-23T03:54:46.1794423Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:10 with 2 nodes. 2022-11-23T03:54:46.1795185Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1795948Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1796170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 0 2022-11-23T03:54:46.1796390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 1 2022-11-23T03:54:46.1796785Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:11 with 2 nodes. 2022-11-23T03:54:46.1797178Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:11 with 2 nodes. 2022-11-23T03:54:46.1797400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 0 2022-11-23T03:54:46.1797671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 1 2022-11-23T03:54:46.1798067Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:12 with 2 nodes. 2022-11-23T03:54:46.1798459Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:12 with 2 nodes. 2022-11-23T03:54:46.1798682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 0 2022-11-23T03:54:46.1798905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 1 2022-11-23T03:54:46.1799286Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:13 with 2 nodes. 2022-11-23T03:54:46.1799675Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:13 with 2 nodes. 2022-11-23T03:54:46.1800479Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1801256Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1802019Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1802251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 0 2022-11-23T03:54:46.1802471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 1 2022-11-23T03:54:46.1802867Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:14 with 2 nodes. 2022-11-23T03:54:46.1803248Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:14 with 2 nodes. 2022-11-23T03:54:46.1804020Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1804244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 0 2022-11-23T03:54:46.1804456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 1 2022-11-23T03:54:46.1804850Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:15 with 2 nodes. 2022-11-23T03:54:46.1805241Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:15 with 2 nodes. 2022-11-23T03:54:46.1805463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 1 2022-11-23T03:54:46.1805686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 0 2022-11-23T03:54:46.1806082Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:16 with 2 nodes. 2022-11-23T03:54:46.1806478Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:16 with 2 nodes. 2022-11-23T03:54:46.1807292Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1808108Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1808868Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1809681Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1810461Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1811216Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1811446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 1 2022-11-23T03:54:46.1811668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 0 2022-11-23T03:54:46.1812062Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:17 with 2 nodes. 2022-11-23T03:54:46.1812441Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:17 with 2 nodes. 2022-11-23T03:54:46.1813205Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1813965Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1814725Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1814952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 1 2022-11-23T03:54:46.1815175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 0 2022-11-23T03:54:46.1815571Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:18 with 2 nodes. 2022-11-23T03:54:46.1815962Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:18 with 2 nodes. 2022-11-23T03:54:46.1816767Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1816990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 1 2022-11-23T03:54:46.1817212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 0 2022-11-23T03:54:46.1817592Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:19 with 2 nodes. 2022-11-23T03:54:46.1817984Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:19 with 2 nodes. 2022-11-23T03:54:46.1818248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:20 to store for rank: 0 2022-11-23T03:54:46.1818650Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:20 with 2 nodes. 2022-11-23T03:54:46.1818874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:20 to store for rank: 1 2022-11-23T03:54:46.1819262Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:20 with 2 nodes. 2022-11-23T03:54:46.1819484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:21 to store for rank: 1 2022-11-23T03:54:46.1819704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:21 to store for rank: 0 2022-11-23T03:54:46.1820097Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:21 with 2 nodes. 2022-11-23T03:54:46.1820492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:21 with 2 nodes. 2022-11-23T03:54:46.1820719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:22 to store for rank: 0 2022-11-23T03:54:46.1820940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:22 to store for rank: 1 2022-11-23T03:54:46.1821334Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:22 with 2 nodes. 2022-11-23T03:54:46.1821721Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:22 with 2 nodes. 2022-11-23T03:54:46.1822480Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1823241Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1823467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:23 to store for rank: 0 2022-11-23T03:54:46.1823688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:23 to store for rank: 1 2022-11-23T03:54:46.1824082Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:23 with 2 nodes. 2022-11-23T03:54:46.1824468Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:23 with 2 nodes. 2022-11-23T03:54:46.1824689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:24 to store for rank: 0 2022-11-23T03:54:46.1824913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:24 to store for rank: 1 2022-11-23T03:54:46.1825364Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:24 with 2 nodes. 2022-11-23T03:54:46.1825753Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:24 with 2 nodes. 2022-11-23T03:54:46.1825975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:25 to store for rank: 1 2022-11-23T03:54:46.1826195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:25 to store for rank: 0 2022-11-23T03:54:46.1826584Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:25 with 2 nodes. 2022-11-23T03:54:46.1826971Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:25 with 2 nodes. 2022-11-23T03:54:46.1827074Z dist init r=0, world=2 2022-11-23T03:54:46.1827225Z dist init r=1, world=2 2022-11-23T03:54:46.1827308Z ok (33.783s) 2022-11-23T03:54:46.1827641Z test_mixture_of_experts_with_delay_before_free_offload_false_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46221 2022-11-23T03:54:46.1827851Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46222 2022-11-23T03:54:46.1828234Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.1828403Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.1828788Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.1828968Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.1829194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.1829577Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.1829743Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.1830129Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.1830309Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.1830533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.1830927Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.1831317Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.1831597Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.1831882Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.1832103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.1832319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.1833374Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.1833477Z warnings.warn( 2022-11-23T03:54:46.1833703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T03:54:46.1834804Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.1834906Z warnings.warn( 2022-11-23T03:54:46.1835130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T03:54:46.1835526Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:54:46.1835919Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:54:46.1836183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T03:54:46.1836411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T03:54:46.1836805Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T03:54:46.1837185Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T03:54:46.1837409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T03:54:46.1837633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T03:54:46.1838026Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T03:54:46.1838417Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T03:54:46.1839175Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1839933Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1840156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0 2022-11-23T03:54:46.1840380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1 2022-11-23T03:54:46.1840773Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 2 nodes. 2022-11-23T03:54:46.1841169Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 2 nodes. 2022-11-23T03:54:46.1841393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0 2022-11-23T03:54:46.1841617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1 2022-11-23T03:54:46.1842010Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 2 nodes. 2022-11-23T03:54:46.1842402Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 2 nodes. 2022-11-23T03:54:46.1842623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0 2022-11-23T03:54:46.1842846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1 2022-11-23T03:54:46.1843294Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 2 nodes. 2022-11-23T03:54:46.1843679Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 2 nodes. 2022-11-23T03:54:46.1844428Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1845182Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1845448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 0 2022-11-23T03:54:46.1845674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 1 2022-11-23T03:54:46.1846068Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:8 with 2 nodes. 2022-11-23T03:54:46.1846457Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:8 with 2 nodes. 2022-11-23T03:54:46.1846677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 0 2022-11-23T03:54:46.1846902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 1 2022-11-23T03:54:46.1847292Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:9 with 2 nodes. 2022-11-23T03:54:46.1847791Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:9 with 2 nodes. 2022-11-23T03:54:46.1848017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 0 2022-11-23T03:54:46.1848238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 1 2022-11-23T03:54:46.1848635Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:10 with 2 nodes. 2022-11-23T03:54:46.1849026Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:10 with 2 nodes. 2022-11-23T03:54:46.1849783Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1850545Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1850771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 0 2022-11-23T03:54:46.1850992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 1 2022-11-23T03:54:46.1851385Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:11 with 2 nodes. 2022-11-23T03:54:46.1851778Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:11 with 2 nodes. 2022-11-23T03:54:46.1851998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 0 2022-11-23T03:54:46.1852286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 1 2022-11-23T03:54:46.1852680Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:12 with 2 nodes. 2022-11-23T03:54:46.1853071Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:12 with 2 nodes. 2022-11-23T03:54:46.1853278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 0 2022-11-23T03:54:46.1853496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 1 2022-11-23T03:54:46.1853891Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:13 with 2 nodes. 2022-11-23T03:54:46.1854282Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:13 with 2 nodes. 2022-11-23T03:54:46.1855114Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1855875Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1856099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 0 2022-11-23T03:54:46.1856321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 1 2022-11-23T03:54:46.1856720Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:14 with 2 nodes. 2022-11-23T03:54:46.1857112Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:14 with 2 nodes. 2022-11-23T03:54:46.1857341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 0 2022-11-23T03:54:46.1857564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 1 2022-11-23T03:54:46.1857961Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:15 with 2 nodes. 2022-11-23T03:54:46.1858351Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:15 with 2 nodes. 2022-11-23T03:54:46.1858577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 0 2022-11-23T03:54:46.1858802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 1 2022-11-23T03:54:46.1859200Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:16 with 2 nodes. 2022-11-23T03:54:46.1859587Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:16 with 2 nodes. 2022-11-23T03:54:46.1860346Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1861100Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1861913Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1862138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 1 2022-11-23T03:54:46.1862358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 0 2022-11-23T03:54:46.1862748Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:17 with 2 nodes. 2022-11-23T03:54:46.1863139Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:17 with 2 nodes. 2022-11-23T03:54:46.1863939Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1864169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 0 2022-11-23T03:54:46.1864393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 1 2022-11-23T03:54:46.1864786Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:18 with 2 nodes. 2022-11-23T03:54:46.1865175Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:18 with 2 nodes. 2022-11-23T03:54:46.1865399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 0 2022-11-23T03:54:46.1865619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 1 2022-11-23T03:54:46.1866019Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:19 with 2 nodes. 2022-11-23T03:54:46.1866406Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:19 with 2 nodes. 2022-11-23T03:54:46.1866628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:20 to store for rank: 0 2022-11-23T03:54:46.1866849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:20 to store for rank: 1 2022-11-23T03:54:46.1867240Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:20 with 2 nodes. 2022-11-23T03:54:46.1867629Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:20 with 2 nodes. 2022-11-23T03:54:46.1868391Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1868617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:21 to store for rank: 1 2022-11-23T03:54:46.1868840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:21 to store for rank: 0 2022-11-23T03:54:46.1869233Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:21 with 2 nodes. 2022-11-23T03:54:46.1869619Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:21 with 2 nodes. 2022-11-23T03:54:46.1869844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:22 to store for rank: 0 2022-11-23T03:54:46.1870067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:22 to store for rank: 1 2022-11-23T03:54:46.1870517Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:22 with 2 nodes. 2022-11-23T03:54:46.1870907Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:22 with 2 nodes. 2022-11-23T03:54:46.1871658Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1871882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:23 to store for rank: 0 2022-11-23T03:54:46.1872089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:23 to store for rank: 1 2022-11-23T03:54:46.1872521Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:23 with 2 nodes. 2022-11-23T03:54:46.1872922Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:23 with 2 nodes. 2022-11-23T03:54:46.1873147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:24 to store for rank: 0 2022-11-23T03:54:46.1873367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:24 to store for rank: 1 2022-11-23T03:54:46.1873757Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:24 with 2 nodes. 2022-11-23T03:54:46.1874147Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:24 with 2 nodes. 2022-11-23T03:54:46.1874906Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1875132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:25 to store for rank: 1 2022-11-23T03:54:46.1875353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:25 to store for rank: 0 2022-11-23T03:54:46.1875745Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:25 with 2 nodes. 2022-11-23T03:54:46.1876132Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:25 with 2 nodes. 2022-11-23T03:54:46.1876235Z dist init r=0, world=2 2022-11-23T03:54:46.1876338Z dist init r=1, world=2 2022-11-23T03:54:46.1876432Z ok (34.486s) 2022-11-23T03:54:46.1876782Z test_mixture_of_experts_with_delay_before_free_offload_false_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46614 2022-11-23T03:54:46.1876996Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46615 2022-11-23T03:54:46.1877373Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.1877539Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.1877924Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.1878102Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.1878325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.1878700Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.1878864Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.1879250Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.1879494Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.1879723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.1880120Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.1880513Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.1880788Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.1881065Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.1881282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.1881544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.1882615Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.1882718Z warnings.warn( 2022-11-23T03:54:46.1882942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T03:54:46.1883985Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.1884089Z warnings.warn( 2022-11-23T03:54:46.1884309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T03:54:46.1884703Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:54:46.1885097Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:54:46.1885319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T03:54:46.1885541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T03:54:46.1885936Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T03:54:46.1886326Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T03:54:46.1886545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T03:54:46.1886763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T03:54:46.1887150Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T03:54:46.1887540Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T03:54:46.1888935Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1889929Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1890155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0 2022-11-23T03:54:46.1890549Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 2 nodes. 2022-11-23T03:54:46.1890770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1 2022-11-23T03:54:46.1891163Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 2 nodes. 2022-11-23T03:54:46.1891445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1 2022-11-23T03:54:46.1891671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0 2022-11-23T03:54:46.1892060Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 2 nodes. 2022-11-23T03:54:46.1892450Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 2 nodes. 2022-11-23T03:54:46.1892671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1 2022-11-23T03:54:46.1893063Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 2 nodes. 2022-11-23T03:54:46.1893284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0 2022-11-23T03:54:46.1893681Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 2 nodes. 2022-11-23T03:54:46.1894443Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1895201Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1895422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 1 2022-11-23T03:54:46.1895812Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:8 with 2 nodes. 2022-11-23T03:54:46.1896037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 0 2022-11-23T03:54:46.1896414Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:8 with 2 nodes. 2022-11-23T03:54:46.1896633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 1 2022-11-23T03:54:46.1896855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 0 2022-11-23T03:54:46.1897244Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:9 with 2 nodes. 2022-11-23T03:54:46.1897632Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:9 with 2 nodes. 2022-11-23T03:54:46.1897857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 1 2022-11-23T03:54:46.1898083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 0 2022-11-23T03:54:46.1898526Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:10 with 2 nodes. 2022-11-23T03:54:46.1898921Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:10 with 2 nodes. 2022-11-23T03:54:46.1899683Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1900479Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1900712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 1 2022-11-23T03:54:46.1900934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 0 2022-11-23T03:54:46.1901328Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:11 with 2 nodes. 2022-11-23T03:54:46.1901718Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:11 with 2 nodes. 2022-11-23T03:54:46.1901940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 1 2022-11-23T03:54:46.1902159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 0 2022-11-23T03:54:46.1902551Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:12 with 2 nodes. 2022-11-23T03:54:46.1902948Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:12 with 2 nodes. 2022-11-23T03:54:46.1903171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 1 2022-11-23T03:54:46.1903391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 0 2022-11-23T03:54:46.1903782Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:13 with 2 nodes. 2022-11-23T03:54:46.1904169Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:13 with 2 nodes. 2022-11-23T03:54:46.1904930Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1905690Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1905914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 1 2022-11-23T03:54:46.1906135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 0 2022-11-23T03:54:46.1906527Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:14 with 2 nodes. 2022-11-23T03:54:46.1906917Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:14 with 2 nodes. 2022-11-23T03:54:46.1907140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 1 2022-11-23T03:54:46.1907521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 0 2022-11-23T03:54:46.1907920Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:15 with 2 nodes. 2022-11-23T03:54:46.1908310Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:15 with 2 nodes. 2022-11-23T03:54:46.1908533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 1 2022-11-23T03:54:46.1908751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 0 2022-11-23T03:54:46.1909142Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:16 with 2 nodes. 2022-11-23T03:54:46.1909572Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:16 with 2 nodes. 2022-11-23T03:54:46.1910338Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1911098Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1911865Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1912092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 1 2022-11-23T03:54:46.1912313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 0 2022-11-23T03:54:46.1912703Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:17 with 2 nodes. 2022-11-23T03:54:46.1913097Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:17 with 2 nodes. 2022-11-23T03:54:46.1913853Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1914076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 0 2022-11-23T03:54:46.1914301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 1 2022-11-23T03:54:46.1914692Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:18 with 2 nodes. 2022-11-23T03:54:46.1915083Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:18 with 2 nodes. 2022-11-23T03:54:46.1915291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 1 2022-11-23T03:54:46.1915512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 0 2022-11-23T03:54:46.1915906Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:19 with 2 nodes. 2022-11-23T03:54:46.1916307Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:19 with 2 nodes. 2022-11-23T03:54:46.1916581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:20 to store for rank: 1 2022-11-23T03:54:46.1916802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:20 to store for rank: 0 2022-11-23T03:54:46.1917197Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:20 with 2 nodes. 2022-11-23T03:54:46.1917589Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:20 with 2 nodes. 2022-11-23T03:54:46.1918348Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1918616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:21 to store for rank: 1 2022-11-23T03:54:46.1918843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:21 to store for rank: 0 2022-11-23T03:54:46.1919239Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:21 with 2 nodes. 2022-11-23T03:54:46.1919627Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:21 with 2 nodes. 2022-11-23T03:54:46.1919850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:22 to store for rank: 1 2022-11-23T03:54:46.1920070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:22 to store for rank: 0 2022-11-23T03:54:46.1920462Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:22 with 2 nodes. 2022-11-23T03:54:46.1920849Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:22 with 2 nodes. 2022-11-23T03:54:46.1921615Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1921838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:23 to store for rank: 1 2022-11-23T03:54:46.1922063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:23 to store for rank: 0 2022-11-23T03:54:46.1922457Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:23 with 2 nodes. 2022-11-23T03:54:46.1922847Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:23 with 2 nodes. 2022-11-23T03:54:46.1923072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:24 to store for rank: 1 2022-11-23T03:54:46.1923300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:24 to store for rank: 0 2022-11-23T03:54:46.1923690Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:24 with 2 nodes. 2022-11-23T03:54:46.1924076Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:24 with 2 nodes. 2022-11-23T03:54:46.1924832Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.1925052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:25 to store for rank: 1 2022-11-23T03:54:46.1925275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:25 to store for rank: 0 2022-11-23T03:54:46.1925736Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:25 with 2 nodes. 2022-11-23T03:54:46.1926127Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:25 with 2 nodes. 2022-11-23T03:54:46.1926230Z dist init r=0, world=2 2022-11-23T03:54:46.1926332Z dist init r=1, world=2 2022-11-23T03:54:46.1926411Z ok (33.273s) 2022-11-23T03:54:46.1926753Z test_mixture_of_experts_with_delay_before_free_offload_true_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47007 2022-11-23T03:54:46.1926963Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47008 2022-11-23T03:54:46.1927339Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.1927553Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.1928082Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.1928262Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.1928489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.1928865Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.1929031Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.1929420Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.1929599Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.1929823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.1930225Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.1930616Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.1930893Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.1931174Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.1931389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.1931603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.1932649Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.1932754Z warnings.warn( 2022-11-23T03:54:46.1932978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T03:54:46.1934011Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.1934114Z warnings.warn( 2022-11-23T03:54:46.1934339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T03:54:46.1934804Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:54:46.1935199Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:54:46.1935320Z File "", line 1, in 2022-11-23T03:54:46.1935519Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1935641Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1935833Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1935976Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1936180Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1936276Z self.run() 2022-11-23T03:54:46.1936470Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1936662Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1937016Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1937143Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1937523Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1937638Z getattr(self, test_name)() 2022-11-23T03:54:46.1938011Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1938104Z fn() 2022-11-23T03:54:46.1938480Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1938597Z test(self, **param_kwargs) 2022-11-23T03:54:46.1938967Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1939090Z return func(*args, **kwargs) 2022-11-23T03:54:46.1939342Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.1939452Z self.run_subtests( 2022-11-23T03:54:46.1939810Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1939965Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1940337Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1940481Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1940866Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1940976Z output = model(*input) 2022-11-23T03:54:46.1941315Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1941451Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1941836Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1942002Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1942381Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1942497Z _lazy_init(state, module) 2022-11-23T03:54:46.1942858Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1942991Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1943335Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1943452Z return func(*args, **kwargs) 2022-11-23T03:54:46.1943904Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1943984Z p_assert( 2022-11-23T03:54:46.1944327Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1944444Z traceback.print_stack() 2022-11-23T03:54:46.1944672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T03:54:46.1944793Z File "", line 1, in 2022-11-23T03:54:46.1944992Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1945124Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1945314Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1945458Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1945661Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1945805Z self.run() 2022-11-23T03:54:46.1946001Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1946138Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1946489Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1946613Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1946984Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1947087Z getattr(self, test_name)() 2022-11-23T03:54:46.1947457Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1947552Z fn() 2022-11-23T03:54:46.1947926Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1948041Z test(self, **param_kwargs) 2022-11-23T03:54:46.1948415Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1948532Z return func(*args, **kwargs) 2022-11-23T03:54:46.1948800Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.1948905Z self.run_subtests( 2022-11-23T03:54:46.1949267Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1949420Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1949791Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1949935Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1950322Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1950435Z output = model(*input) 2022-11-23T03:54:46.1950773Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1950906Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1951287Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1951454Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1951816Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1951930Z _lazy_init(state, module) 2022-11-23T03:54:46.1952287Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1952420Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1952766Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1952947Z return func(*args, **kwargs) 2022-11-23T03:54:46.1953335Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1953429Z p_assert( 2022-11-23T03:54:46.1953769Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1953885Z traceback.print_stack() 2022-11-23T03:54:46.1954111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T03:54:46.1954504Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T03:54:46.1954893Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T03:54:46.1955014Z File "", line 1, in 2022-11-23T03:54:46.1955260Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1955399Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1955592Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1955733Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1955924Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1956021Z self.run() 2022-11-23T03:54:46.1956218Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1956355Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1956707Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1956835Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1957205Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1957330Z getattr(self, test_name)() 2022-11-23T03:54:46.1957696Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1957788Z fn() 2022-11-23T03:54:46.1958161Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1958281Z test(self, **param_kwargs) 2022-11-23T03:54:46.1958645Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1958762Z return func(*args, **kwargs) 2022-11-23T03:54:46.1959026Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.1959134Z self.run_subtests( 2022-11-23T03:54:46.1959492Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1959637Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1960008Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1960152Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1960534Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1960646Z output = model(*input) 2022-11-23T03:54:46.1960979Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1961112Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1961495Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1961660Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1962040Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1962216Z _lazy_init(state, module) 2022-11-23T03:54:46.1962576Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1962709Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1963055Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1963172Z return func(*args, **kwargs) 2022-11-23T03:54:46.1963556Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1963650Z p_assert( 2022-11-23T03:54:46.1963990Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1964105Z traceback.print_stack() 2022-11-23T03:54:46.1964367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T03:54:46.1964494Z File "", line 1, in 2022-11-23T03:54:46.1964695Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1964829Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1965022Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1965163Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1965366Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1965462Z self.run() 2022-11-23T03:54:46.1965655Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1965794Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1966143Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1966269Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1966644Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1966761Z getattr(self, test_name)() 2022-11-23T03:54:46.1967129Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1967220Z fn() 2022-11-23T03:54:46.1967580Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1967806Z test(self, **param_kwargs) 2022-11-23T03:54:46.1968433Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1968654Z return func(*args, **kwargs) 2022-11-23T03:54:46.1969052Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.1969162Z self.run_subtests( 2022-11-23T03:54:46.1969543Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1969697Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1970066Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1970196Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1970584Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1970696Z output = model(*input) 2022-11-23T03:54:46.1971032Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1971164Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1971550Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1971818Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1972197Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1972310Z _lazy_init(state, module) 2022-11-23T03:54:46.1972670Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1972805Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1973151Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1973267Z return func(*args, **kwargs) 2022-11-23T03:54:46.1973654Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1973750Z p_assert( 2022-11-23T03:54:46.1974144Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1974269Z traceback.print_stack() 2022-11-23T03:54:46.1974496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T03:54:46.1974898Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T03:54:46.1975276Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T03:54:46.1975401Z File "", line 1, in 2022-11-23T03:54:46.1975601Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1975737Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1975929Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1976073Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1976279Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1976378Z self.run() 2022-11-23T03:54:46.1976572Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1976708Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1977056Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1977187Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1977560Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1977676Z getattr(self, test_name)() 2022-11-23T03:54:46.1978045Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1978135Z fn() 2022-11-23T03:54:46.1978508Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1978617Z test(self, **param_kwargs) 2022-11-23T03:54:46.1978984Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1979099Z return func(*args, **kwargs) 2022-11-23T03:54:46.1979364Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.1979470Z self.run_subtests( 2022-11-23T03:54:46.1979830Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1979983Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1980362Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1980504Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1980897Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1981069Z output = model(*input) 2022-11-23T03:54:46.1981411Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1981544Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1981930Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1982096Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1982471Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1982586Z _lazy_init(state, module) 2022-11-23T03:54:46.1982945Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1983079Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1983457Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1983583Z return func(*args, **kwargs) 2022-11-23T03:54:46.1983971Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1984066Z p_assert( 2022-11-23T03:54:46.1984408Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1984524Z traceback.print_stack() 2022-11-23T03:54:46.1984750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1 2022-11-23T03:54:46.1984872Z File "", line 1, in 2022-11-23T03:54:46.1985071Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1985204Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1985397Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1985550Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1985754Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1985854Z self.run() 2022-11-23T03:54:46.1986047Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1986183Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1986531Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1986642Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1987013Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1987130Z getattr(self, test_name)() 2022-11-23T03:54:46.1987495Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1987586Z fn() 2022-11-23T03:54:46.1987964Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1988080Z test(self, **param_kwargs) 2022-11-23T03:54:46.1988446Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1988564Z return func(*args, **kwargs) 2022-11-23T03:54:46.1988825Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.1988932Z self.run_subtests( 2022-11-23T03:54:46.1989291Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1989443Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.1989815Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.1990025Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.1990411Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.1990521Z output = model(*input) 2022-11-23T03:54:46.1990857Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.1990977Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.1991367Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.1991536Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.1991909Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.1992023Z _lazy_init(state, module) 2022-11-23T03:54:46.1992444Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.1992584Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.1992936Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.1993054Z return func(*args, **kwargs) 2022-11-23T03:54:46.1993436Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.1993531Z p_assert( 2022-11-23T03:54:46.1993873Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.1993992Z traceback.print_stack() 2022-11-23T03:54:46.1994219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0 2022-11-23T03:54:46.1994615Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 2 nodes. 2022-11-23T03:54:46.1995009Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 2 nodes. 2022-11-23T03:54:46.1995136Z File "", line 1, in 2022-11-23T03:54:46.1995336Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.1995470Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.1995649Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.1995795Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.1995999Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.1996095Z self.run() 2022-11-23T03:54:46.1996294Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.1996432Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.1996776Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.1996907Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.1997276Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.1997394Z getattr(self, test_name)() 2022-11-23T03:54:46.1997760Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.1997853Z fn() 2022-11-23T03:54:46.1998226Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.1998343Z test(self, **param_kwargs) 2022-11-23T03:54:46.1998708Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.1998824Z return func(*args, **kwargs) 2022-11-23T03:54:46.1999088Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.1999246Z self.run_subtests( 2022-11-23T03:54:46.1999610Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.1999766Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2000138Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2000283Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2000665Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2000773Z output = model(*input) 2022-11-23T03:54:46.2001107Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2001242Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2001669Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2001842Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2002221Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2002334Z _lazy_init(state, module) 2022-11-23T03:54:46.2002694Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2002831Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2003178Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2003295Z return func(*args, **kwargs) 2022-11-23T03:54:46.2003680Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2003774Z p_assert( 2022-11-23T03:54:46.2004104Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2004224Z traceback.print_stack() 2022-11-23T03:54:46.2004447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1 2022-11-23T03:54:46.2004569Z File "", line 1, in 2022-11-23T03:54:46.2004767Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2004901Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2005091Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2005234Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2005437Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2005533Z self.run() 2022-11-23T03:54:46.2005727Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2005863Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2006215Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2006339Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2006712Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2006828Z getattr(self, test_name)() 2022-11-23T03:54:46.2007179Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2007271Z fn() 2022-11-23T03:54:46.2007641Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2007891Z test(self, **param_kwargs) 2022-11-23T03:54:46.2008266Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2008381Z return func(*args, **kwargs) 2022-11-23T03:54:46.2008723Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2008828Z self.run_subtests( 2022-11-23T03:54:46.2009194Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2009347Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2009719Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2009863Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2010246Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2010354Z output = model(*input) 2022-11-23T03:54:46.2010687Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2010870Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2011260Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2011427Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2011802Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2011903Z _lazy_init(state, module) 2022-11-23T03:54:46.2012263Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2012396Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2012743Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2012859Z return func(*args, **kwargs) 2022-11-23T03:54:46.2013255Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2013354Z p_assert( 2022-11-23T03:54:46.2013694Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2013811Z traceback.print_stack() 2022-11-23T03:54:46.2014034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0 2022-11-23T03:54:46.2014427Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 2 nodes. 2022-11-23T03:54:46.2014818Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 2 nodes. 2022-11-23T03:54:46.2014940Z File "", line 1, in 2022-11-23T03:54:46.2015140Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2015272Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2015468Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2015613Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2015815Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2015897Z self.run() 2022-11-23T03:54:46.2016092Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2016228Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2016574Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2016700Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2017070Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2017186Z getattr(self, test_name)() 2022-11-23T03:54:46.2017552Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2017707Z fn() 2022-11-23T03:54:46.2018092Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2018212Z test(self, **param_kwargs) 2022-11-23T03:54:46.2018576Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2018694Z return func(*args, **kwargs) 2022-11-23T03:54:46.2018955Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2019060Z self.run_subtests( 2022-11-23T03:54:46.2019418Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2019572Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2019942Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2020119Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2020508Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2020618Z output = model(*input) 2022-11-23T03:54:46.2020952Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2021085Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2021471Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2021639Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2022014Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2022132Z _lazy_init(state, module) 2022-11-23T03:54:46.2022493Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2022630Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2022979Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2023095Z return func(*args, **kwargs) 2022-11-23T03:54:46.2023477Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2023572Z p_assert( 2022-11-23T03:54:46.2023911Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2024028Z traceback.print_stack() 2022-11-23T03:54:46.2024254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1 2022-11-23T03:54:46.2024634Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 2 nodes. 2022-11-23T03:54:46.2024761Z File "", line 1, in 2022-11-23T03:54:46.2024961Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2025095Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2025292Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2025434Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2025638Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2025734Z self.run() 2022-11-23T03:54:46.2025930Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2026066Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2026415Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2026539Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2026915Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2027094Z getattr(self, test_name)() 2022-11-23T03:54:46.2027464Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2027557Z fn() 2022-11-23T03:54:46.2027931Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2028033Z test(self, **param_kwargs) 2022-11-23T03:54:46.2028394Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2028509Z return func(*args, **kwargs) 2022-11-23T03:54:46.2028770Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2028874Z self.run_subtests( 2022-11-23T03:54:46.2029280Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2029441Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2029817Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2029962Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2030346Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2030455Z output = model(*input) 2022-11-23T03:54:46.2030794Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2030927Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2031312Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2031479Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2031862Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2031977Z _lazy_init(state, module) 2022-11-23T03:54:46.2032337Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2032473Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2032804Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2032921Z return func(*args, **kwargs) 2022-11-23T03:54:46.2033308Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2033403Z p_assert( 2022-11-23T03:54:46.2033746Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2033862Z traceback.print_stack() 2022-11-23T03:54:46.2034098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0 2022-11-23T03:54:46.2034497Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 2 nodes. 2022-11-23T03:54:46.2034621Z File "", line 1, in 2022-11-23T03:54:46.2034822Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2034955Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2035147Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2035291Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2035493Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2035592Z self.run() 2022-11-23T03:54:46.2035784Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2035924Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2036319Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2036443Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2036816Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2036933Z getattr(self, test_name)() 2022-11-23T03:54:46.2037297Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2037388Z fn() 2022-11-23T03:54:46.2037761Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2037877Z test(self, **param_kwargs) 2022-11-23T03:54:46.2038247Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2038366Z return func(*args, **kwargs) 2022-11-23T03:54:46.2038675Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2038784Z self.run_subtests( 2022-11-23T03:54:46.2039150Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2039305Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2039679Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2039827Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2040215Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2040326Z output = model(*input) 2022-11-23T03:54:46.2040645Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2040787Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2041173Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2041344Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2041719Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2041832Z _lazy_init(state, module) 2022-11-23T03:54:46.2042193Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2042326Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2042674Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2042792Z return func(*args, **kwargs) 2022-11-23T03:54:46.2043181Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2043280Z p_assert( 2022-11-23T03:54:46.2043619Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2043737Z traceback.print_stack() 2022-11-23T03:54:46.2043965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 1 2022-11-23T03:54:46.2044089Z File "", line 1, in 2022-11-23T03:54:46.2044286Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2044420Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2044599Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2044742Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2044945Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2045041Z self.run() 2022-11-23T03:54:46.2045302Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2045439Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2045790Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2045917Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2046290Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2046410Z getattr(self, test_name)() 2022-11-23T03:54:46.2046777Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2046869Z fn() 2022-11-23T03:54:46.2047244Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2047360Z test(self, **param_kwargs) 2022-11-23T03:54:46.2047891Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2048077Z return func(*args, **kwargs) 2022-11-23T03:54:46.2048589Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2048682Z self.run_subtests( 2022-11-23T03:54:46.2049064Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2049218Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2049589Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2049734Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2050116Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2050226Z output = model(*input) 2022-11-23T03:54:46.2050567Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2050699Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2051084Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2051250Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2051623Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2051735Z _lazy_init(state, module) 2022-11-23T03:54:46.2052091Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2052225Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2052573Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2052696Z return func(*args, **kwargs) 2022-11-23T03:54:46.2053081Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2053182Z p_assert( 2022-11-23T03:54:46.2053508Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2053623Z traceback.print_stack() 2022-11-23T03:54:46.2053850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 0 2022-11-23T03:54:46.2054242Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:8 with 2 nodes. 2022-11-23T03:54:46.2054632Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:8 with 2 nodes. 2022-11-23T03:54:46.2054753Z File "", line 1, in 2022-11-23T03:54:46.2054961Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2055179Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2055371Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2055513Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2055717Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2055813Z self.run() 2022-11-23T03:54:46.2056006Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2056143Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2056494Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2056622Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2056992Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2057107Z getattr(self, test_name)() 2022-11-23T03:54:46.2057532Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2057627Z fn() 2022-11-23T03:54:46.2058005Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2058121Z test(self, **param_kwargs) 2022-11-23T03:54:46.2058489Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2058606Z return func(*args, **kwargs) 2022-11-23T03:54:46.2058871Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2058976Z self.run_subtests( 2022-11-23T03:54:46.2059334Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2059492Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2059865Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2060009Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2060390Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2060501Z output = model(*input) 2022-11-23T03:54:46.2060832Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2060964Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2061349Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2061516Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2061879Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2061995Z _lazy_init(state, module) 2022-11-23T03:54:46.2062359Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2062499Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2062846Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2062963Z return func(*args, **kwargs) 2022-11-23T03:54:46.2063350Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2063443Z p_assert( 2022-11-23T03:54:46.2063782Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2063898Z traceback.print_stack() 2022-11-23T03:54:46.2064121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 0 2022-11-23T03:54:46.2064326Z File "", line 1, in 2022-11-23T03:54:46.2064531Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2064663Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2064855Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2064996Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2065198Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2065279Z self.run() 2022-11-23T03:54:46.2065474Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2065610Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2065960Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2066086Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2066504Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2066625Z getattr(self, test_name)() 2022-11-23T03:54:46.2066998Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2067087Z fn() 2022-11-23T03:54:46.2067460Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2067577Z test(self, **param_kwargs) 2022-11-23T03:54:46.2067943Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2068063Z return func(*args, **kwargs) 2022-11-23T03:54:46.2068326Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2068431Z self.run_subtests( 2022-11-23T03:54:46.2068796Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2068954Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2069325Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2069456Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2069840Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2069952Z output = model(*input) 2022-11-23T03:54:46.2070287Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2070423Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2070806Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2070979Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2071358Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2071471Z _lazy_init(state, module) 2022-11-23T03:54:46.2071832Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2071967Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2072311Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2072427Z return func(*args, **kwargs) 2022-11-23T03:54:46.2072813Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2072907Z p_assert( 2022-11-23T03:54:46.2073245Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2073360Z traceback.print_stack() 2022-11-23T03:54:46.2073648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 1 2022-11-23T03:54:46.2074047Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:9 with 2 nodes. 2022-11-23T03:54:46.2074427Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:9 with 2 nodes. 2022-11-23T03:54:46.2074550Z File "", line 1, in 2022-11-23T03:54:46.2074751Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2074888Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2075081Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2075223Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2075427Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2075528Z self.run() 2022-11-23T03:54:46.2075768Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2075907Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2076256Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2076381Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2076754Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2076871Z getattr(self, test_name)() 2022-11-23T03:54:46.2077237Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2077328Z fn() 2022-11-23T03:54:46.2077704Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2077806Z test(self, **param_kwargs) 2022-11-23T03:54:46.2078179Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2078295Z return func(*args, **kwargs) 2022-11-23T03:54:46.2078554Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2078658Z self.run_subtests( 2022-11-23T03:54:46.2079016Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2079176Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2079546Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2079691Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2080072Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2080189Z output = model(*input) 2022-11-23T03:54:46.2080520Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2080652Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2081038Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2081206Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2081583Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2081700Z _lazy_init(state, module) 2022-11-23T03:54:46.2082058Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2082191Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2082521Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2082706Z return func(*args, **kwargs) 2022-11-23T03:54:46.2083095Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2083190Z p_assert( 2022-11-23T03:54:46.2083527Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2083645Z traceback.print_stack() 2022-11-23T03:54:46.2083874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 1 2022-11-23T03:54:46.2083996Z File "", line 1, in 2022-11-23T03:54:46.2084195Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2084330Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2084522Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2084665Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2084916Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2085016Z self.run() 2022-11-23T03:54:46.2085212Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2085348Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2085682Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2085806Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2086177Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2086293Z getattr(self, test_name)() 2022-11-23T03:54:46.2086658Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2086748Z fn() 2022-11-23T03:54:46.2087125Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2087246Z test(self, **param_kwargs) 2022-11-23T03:54:46.2087613Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2087807Z return func(*args, **kwargs) 2022-11-23T03:54:46.2088115Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2088220Z self.run_subtests( 2022-11-23T03:54:46.2088591Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2088744Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2089114Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2089256Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2089647Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2089758Z output = model(*input) 2022-11-23T03:54:46.2090091Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2090210Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2090592Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2090759Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2091132Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2091245Z _lazy_init(state, module) 2022-11-23T03:54:46.2091600Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2091815Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2092164Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2092282Z return func(*args, **kwargs) 2022-11-23T03:54:46.2092666Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2092761Z p_assert( 2022-11-23T03:54:46.2093105Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2093220Z traceback.print_stack() 2022-11-23T03:54:46.2093448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 0 2022-11-23T03:54:46.2093847Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:10 with 2 nodes. 2022-11-23T03:54:46.2094293Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:10 with 2 nodes. 2022-11-23T03:54:46.2094425Z File "", line 1, in 2022-11-23T03:54:46.2094627Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2094747Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2094940Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2095083Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2095292Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2095387Z self.run() 2022-11-23T03:54:46.2095582Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2095722Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2096071Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2096197Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2096573Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2096689Z getattr(self, test_name)() 2022-11-23T03:54:46.2097054Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2097144Z fn() 2022-11-23T03:54:46.2097513Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2097629Z test(self, **param_kwargs) 2022-11-23T03:54:46.2097994Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2098109Z return func(*args, **kwargs) 2022-11-23T03:54:46.2098374Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2098465Z self.run_subtests( 2022-11-23T03:54:46.2098831Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2098988Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2099360Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2099504Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2099888Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2099999Z output = model(*input) 2022-11-23T03:54:46.2100332Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2100466Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2100850Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2101082Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2101463Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2101578Z _lazy_init(state, module) 2022-11-23T03:54:46.2101935Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2102068Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2102411Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2102530Z return func(*args, **kwargs) 2022-11-23T03:54:46.2102917Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2102997Z p_assert( 2022-11-23T03:54:46.2103340Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2103523Z traceback.print_stack() 2022-11-23T03:54:46.2103753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 1 2022-11-23T03:54:46.2103874Z File "", line 1, in 2022-11-23T03:54:46.2104075Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2104209Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2104402Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2104548Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2104753Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2104847Z self.run() 2022-11-23T03:54:46.2105039Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2105174Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2105528Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2105656Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2106029Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2106149Z getattr(self, test_name)() 2022-11-23T03:54:46.2106502Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2106591Z fn() 2022-11-23T03:54:46.2106964Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2107081Z test(self, **param_kwargs) 2022-11-23T03:54:46.2107443Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2107559Z return func(*args, **kwargs) 2022-11-23T03:54:46.2107826Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2107936Z self.run_subtests( 2022-11-23T03:54:46.2108297Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2108453Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2108823Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2108968Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2109351Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2109464Z output = model(*input) 2022-11-23T03:54:46.2109797Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2109933Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2110391Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2110558Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2110931Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2111031Z _lazy_init(state, module) 2022-11-23T03:54:46.2111387Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2111521Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2111864Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2111983Z return func(*args, **kwargs) 2022-11-23T03:54:46.2112367Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2112464Z p_assert( 2022-11-23T03:54:46.2112847Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2112972Z traceback.print_stack() 2022-11-23T03:54:46.2113199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 0 2022-11-23T03:54:46.2113603Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:11 with 2 nodes. 2022-11-23T03:54:46.2113999Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:11 with 2 nodes. 2022-11-23T03:54:46.2114122Z File "", line 1, in 2022-11-23T03:54:46.2114322Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2114458Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2114650Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2114801Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2115005Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2115087Z self.run() 2022-11-23T03:54:46.2115279Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2115415Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2115761Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2115890Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2116262Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2116377Z getattr(self, test_name)() 2022-11-23T03:54:46.2116747Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2116837Z fn() 2022-11-23T03:54:46.2117211Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2117334Z test(self, **param_kwargs) 2022-11-23T03:54:46.2117699Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2117815Z return func(*args, **kwargs) 2022-11-23T03:54:46.2118076Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2118185Z self.run_subtests( 2022-11-23T03:54:46.2118542Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2118695Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2119052Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2119198Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2119652Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2119765Z output = model(*input) 2022-11-23T03:54:46.2120100Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2120233Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2120617Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2120781Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2121156Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2121269Z _lazy_init(state, module) 2022-11-23T03:54:46.2121628Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2121809Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2122161Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2122278Z return func(*args, **kwargs) 2022-11-23T03:54:46.2122663Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2122759Z p_assert( 2022-11-23T03:54:46.2123102Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2123220Z traceback.print_stack() 2022-11-23T03:54:46.2123450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 1 2022-11-23T03:54:46.2123557Z File "", line 1, in 2022-11-23T03:54:46.2123756Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2123895Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2124091Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2124233Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2124436Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2124535Z self.run() 2022-11-23T03:54:46.2124729Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2124864Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2125209Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2125334Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2125705Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2125819Z getattr(self, test_name)() 2022-11-23T03:54:46.2126187Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2126280Z fn() 2022-11-23T03:54:46.2126654Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2126756Z test(self, **param_kwargs) 2022-11-23T03:54:46.2127121Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2127239Z return func(*args, **kwargs) 2022-11-23T03:54:46.2127505Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2127610Z self.run_subtests( 2022-11-23T03:54:46.2128105Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2128260Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2128637Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2128854Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2129241Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2129349Z output = model(*input) 2022-11-23T03:54:46.2129683Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2129817Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2130196Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2130358Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2130732Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2130845Z _lazy_init(state, module) 2022-11-23T03:54:46.2131272Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2131411Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2131744Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2131865Z return func(*args, **kwargs) 2022-11-23T03:54:46.2132249Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2132344Z p_assert( 2022-11-23T03:54:46.2132681Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2132798Z traceback.print_stack() 2022-11-23T03:54:46.2133023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 0 2022-11-23T03:54:46.2133428Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:12 with 2 nodes. 2022-11-23T03:54:46.2133824Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:12 with 2 nodes. 2022-11-23T03:54:46.2133945Z File "", line 1, in 2022-11-23T03:54:46.2134147Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2134280Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2134472Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2134615Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2134818Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2134914Z self.run() 2022-11-23T03:54:46.2135105Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2135240Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2135580Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2135708Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2136075Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2136190Z getattr(self, test_name)() 2022-11-23T03:54:46.2136554Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2136644Z fn() 2022-11-23T03:54:46.2137017Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2137133Z test(self, **param_kwargs) 2022-11-23T03:54:46.2137494Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2137611Z return func(*args, **kwargs) 2022-11-23T03:54:46.2137877Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2138048Z self.run_subtests( 2022-11-23T03:54:46.2138413Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2138564Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2138936Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2139080Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2139463Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2139576Z output = model(*input) 2022-11-23T03:54:46.2139895Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2140029Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2140462Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2140630Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2141006Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2141122Z _lazy_init(state, module) 2022-11-23T03:54:46.2141481Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2141617Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2141962Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2142077Z return func(*args, **kwargs) 2022-11-23T03:54:46.2142460Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2142561Z p_assert( 2022-11-23T03:54:46.2142901Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2143017Z traceback.print_stack() 2022-11-23T03:54:46.2143241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 1 2022-11-23T03:54:46.2143364Z File "", line 1, in 2022-11-23T03:54:46.2143570Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2143704Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2143882Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2144025Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2144228Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2144324Z self.run() 2022-11-23T03:54:46.2144519Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2144658Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2145006Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2145132Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2145502Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2145620Z getattr(self, test_name)() 2022-11-23T03:54:46.2145985Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2146079Z fn() 2022-11-23T03:54:46.2146451Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2146569Z test(self, **param_kwargs) 2022-11-23T03:54:46.2146942Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2147119Z return func(*args, **kwargs) 2022-11-23T03:54:46.2147385Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2147476Z self.run_subtests( 2022-11-23T03:54:46.2147840Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2147993Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2148364Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2148508Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2148891Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2149003Z output = model(*input) 2022-11-23T03:54:46.2149383Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2149524Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2149910Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2150082Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2150459Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2150574Z _lazy_init(state, module) 2022-11-23T03:54:46.2150935Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2151068Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2151410Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2151529Z return func(*args, **kwargs) 2022-11-23T03:54:46.2151918Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2152012Z p_assert( 2022-11-23T03:54:46.2152337Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2152453Z traceback.print_stack() 2022-11-23T03:54:46.2152681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 0 2022-11-23T03:54:46.2153078Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:13 with 2 nodes. 2022-11-23T03:54:46.2153476Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:13 with 2 nodes. 2022-11-23T03:54:46.2153597Z File "", line 1, in 2022-11-23T03:54:46.2153796Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2153936Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2154125Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2154270Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2154473Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2154568Z self.run() 2022-11-23T03:54:46.2154760Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2154900Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2155247Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2155372Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2155745Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2155848Z getattr(self, test_name)() 2022-11-23T03:54:46.2156218Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2156369Z fn() 2022-11-23T03:54:46.2156742Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2156858Z test(self, **param_kwargs) 2022-11-23T03:54:46.2157225Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2157340Z return func(*args, **kwargs) 2022-11-23T03:54:46.2157601Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2157706Z self.run_subtests( 2022-11-23T03:54:46.2158068Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2158219Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2158637Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2158787Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2159173Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2159284Z output = model(*input) 2022-11-23T03:54:46.2159616Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2159751Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2160136Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2160302Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2160663Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2160780Z _lazy_init(state, module) 2022-11-23T03:54:46.2161138Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2161270Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2161615Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2161736Z return func(*args, **kwargs) 2022-11-23T03:54:46.2162123Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2162216Z p_assert( 2022-11-23T03:54:46.2162556Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2162673Z traceback.print_stack() 2022-11-23T03:54:46.2162899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 1 2022-11-23T03:54:46.2163021Z File "", line 1, in 2022-11-23T03:54:46.2163224Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2163356Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2163547Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2163690Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2163892Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2163974Z self.run() 2022-11-23T03:54:46.2164169Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2164307Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2164653Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2164777Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2165153Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2165330Z getattr(self, test_name)() 2022-11-23T03:54:46.2165700Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2165791Z fn() 2022-11-23T03:54:46.2166163Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2166280Z test(self, **param_kwargs) 2022-11-23T03:54:46.2166647Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2166764Z return func(*args, **kwargs) 2022-11-23T03:54:46.2167026Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2167131Z self.run_subtests( 2022-11-23T03:54:46.2167534Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2167771Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2168371Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2168611Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2169044Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2169156Z output = model(*input) 2022-11-23T03:54:46.2169488Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2169620Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2170005Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2170170Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2170551Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2170665Z _lazy_init(state, module) 2022-11-23T03:54:46.2171022Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2171154Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2171497Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2171614Z return func(*args, **kwargs) 2022-11-23T03:54:46.2171998Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2172094Z p_assert( 2022-11-23T03:54:46.2172433Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2172550Z traceback.print_stack() 2022-11-23T03:54:46.2172783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 0 2022-11-23T03:54:46.2173166Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:14 with 2 nodes. 2022-11-23T03:54:46.2173561Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:14 with 2 nodes. 2022-11-23T03:54:46.2173787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 1 2022-11-23T03:54:46.2174009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 0 2022-11-23T03:54:46.2174401Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:15 with 2 nodes. 2022-11-23T03:54:46.2174793Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:15 with 2 nodes. 2022-11-23T03:54:46.2175018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 1 2022-11-23T03:54:46.2175326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 0 2022-11-23T03:54:46.2175724Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:16 with 2 nodes. 2022-11-23T03:54:46.2176116Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:16 with 2 nodes. 2022-11-23T03:54:46.2176895Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2177124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 1 2022-11-23T03:54:46.2177392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 0 2022-11-23T03:54:46.2177796Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:17 with 2 nodes. 2022-11-23T03:54:46.2178188Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:17 with 2 nodes. 2022-11-23T03:54:46.2178413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 1 2022-11-23T03:54:46.2178640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 0 2022-11-23T03:54:46.2179034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:18 with 2 nodes. 2022-11-23T03:54:46.2179423Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:18 with 2 nodes. 2022-11-23T03:54:46.2180196Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2180970Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2181734Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2181964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 1 2022-11-23T03:54:46.2182191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 0 2022-11-23T03:54:46.2182584Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:19 with 2 nodes. 2022-11-23T03:54:46.2182978Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:19 with 2 nodes. 2022-11-23T03:54:46.2183739Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2183964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:20 to store for rank: 1 2022-11-23T03:54:46.2184189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:20 to store for rank: 0 2022-11-23T03:54:46.2184633Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:20 with 2 nodes. 2022-11-23T03:54:46.2185025Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:20 with 2 nodes. 2022-11-23T03:54:46.2185787Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2186596Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2186827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:21 to store for rank: 1 2022-11-23T03:54:46.2187050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:21 to store for rank: 0 2022-11-23T03:54:46.2187444Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:21 with 2 nodes. 2022-11-23T03:54:46.2187839Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:21 with 2 nodes. 2022-11-23T03:54:46.2188065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:22 to store for rank: 1 2022-11-23T03:54:46.2188286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:22 to store for rank: 0 2022-11-23T03:54:46.2188678Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:22 with 2 nodes. 2022-11-23T03:54:46.2189080Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:22 with 2 nodes. 2022-11-23T03:54:46.2189302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:23 to store for rank: 1 2022-11-23T03:54:46.2189522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:23 to store for rank: 0 2022-11-23T03:54:46.2189912Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:23 with 2 nodes. 2022-11-23T03:54:46.2190302Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:23 with 2 nodes. 2022-11-23T03:54:46.2190527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:24 to store for rank: 1 2022-11-23T03:54:46.2190752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:24 to store for rank: 0 2022-11-23T03:54:46.2191132Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:24 with 2 nodes. 2022-11-23T03:54:46.2191526Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:24 with 2 nodes. 2022-11-23T03:54:46.2192288Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2193066Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2193292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:25 to store for rank: 1 2022-11-23T03:54:46.2193553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:25 to store for rank: 0 2022-11-23T03:54:46.2193948Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:25 with 2 nodes. 2022-11-23T03:54:46.2194342Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:25 with 2 nodes. 2022-11-23T03:54:46.2194565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:26 to store for rank: 1 2022-11-23T03:54:46.2194787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:26 to store for rank: 0 2022-11-23T03:54:46.2195180Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:26 with 2 nodes. 2022-11-23T03:54:46.2195724Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:26 with 2 nodes. 2022-11-23T03:54:46.2195956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:27 to store for rank: 1 2022-11-23T03:54:46.2196180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:27 to store for rank: 0 2022-11-23T03:54:46.2196575Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:27 with 2 nodes. 2022-11-23T03:54:46.2196963Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:27 with 2 nodes. 2022-11-23T03:54:46.2197188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:28 to store for rank: 1 2022-11-23T03:54:46.2197412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:28 to store for rank: 0 2022-11-23T03:54:46.2197804Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:28 with 2 nodes. 2022-11-23T03:54:46.2198199Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:28 with 2 nodes. 2022-11-23T03:54:46.2198960Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2199730Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2199956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:29 to store for rank: 1 2022-11-23T03:54:46.2200182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:29 to store for rank: 0 2022-11-23T03:54:46.2200576Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:29 with 2 nodes. 2022-11-23T03:54:46.2200966Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:29 with 2 nodes. 2022-11-23T03:54:46.2201190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:30 to store for rank: 1 2022-11-23T03:54:46.2201413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:30 to store for rank: 0 2022-11-23T03:54:46.2201806Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:30 with 2 nodes. 2022-11-23T03:54:46.2202195Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:30 with 2 nodes. 2022-11-23T03:54:46.2202420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:31 to store for rank: 1 2022-11-23T03:54:46.2202713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:31 to store for rank: 0 2022-11-23T03:54:46.2203113Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:31 with 2 nodes. 2022-11-23T03:54:46.2203501Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:31 with 2 nodes. 2022-11-23T03:54:46.2203728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:32 to store for rank: 1 2022-11-23T03:54:46.2204119Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:32 with 2 nodes. 2022-11-23T03:54:46.2204347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:32 to store for rank: 0 2022-11-23T03:54:46.2204780Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:32 with 2 nodes. 2022-11-23T03:54:46.2205556Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2205781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:33 to store for rank: 1 2022-11-23T03:54:46.2206538Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2206766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:33 to store for rank: 0 2022-11-23T03:54:46.2207163Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:33 with 2 nodes. 2022-11-23T03:54:46.2207541Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:33 with 2 nodes. 2022-11-23T03:54:46.2207827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:34 to store for rank: 1 2022-11-23T03:54:46.2208049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:34 to store for rank: 0 2022-11-23T03:54:46.2208443Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:34 with 2 nodes. 2022-11-23T03:54:46.2208837Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:34 with 2 nodes. 2022-11-23T03:54:46.2209598Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2210363Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2211123Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2211348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:35 to store for rank: 1 2022-11-23T03:54:46.2211577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:35 to store for rank: 0 2022-11-23T03:54:46.2212030Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:35 with 2 nodes. 2022-11-23T03:54:46.2212421Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:35 with 2 nodes. 2022-11-23T03:54:46.2213180Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2213406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:36 to store for rank: 1 2022-11-23T03:54:46.2213628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:36 to store for rank: 0 2022-11-23T03:54:46.2214068Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:36 with 2 nodes. 2022-11-23T03:54:46.2214468Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:36 with 2 nodes. 2022-11-23T03:54:46.2215229Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2215986Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2216212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:37 to store for rank: 1 2022-11-23T03:54:46.2216438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:37 to store for rank: 0 2022-11-23T03:54:46.2216831Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:37 with 2 nodes. 2022-11-23T03:54:46.2217221Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:37 with 2 nodes. 2022-11-23T03:54:46.2217326Z dist init r=0, world=2 2022-11-23T03:54:46.2217640Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.2217951Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.2218261Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.2218567Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.2218870Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.2219171Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.2219470Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.2219769Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.2220123Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.2220421Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.2220724Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.2221021Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.2221125Z dist init r=1, world=2 2022-11-23T03:54:46.2221439Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.2221782Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.2222091Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.2222394Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.2222695Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.2222994Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.2223297Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.2223600Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.2223901Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.2224199Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.2224499Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.2224801Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.2224902Z ok (40.594s) 2022-11-23T03:54:46.2225240Z test_mixture_of_experts_with_delay_before_free_offload_true_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47424 2022-11-23T03:54:46.2225452Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47425 2022-11-23T03:54:46.2225845Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.2226013Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.2226401Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.2226581Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.2226791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.2227226Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.2227393Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.2227780Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.2227962Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.2228188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.2228583Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.2228977Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.2229254Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.2229578Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.2229798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.2230015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.2231070Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.2231172Z warnings.warn( 2022-11-23T03:54:46.2231391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T03:54:46.2232436Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.2232534Z warnings.warn( 2022-11-23T03:54:46.2232753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T03:54:46.2233143Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:54:46.2233532Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:54:46.2233653Z File "", line 1, in 2022-11-23T03:54:46.2233852Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2233983Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2234170Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2234307Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2234505Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2234587Z self.run() 2022-11-23T03:54:46.2234776Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2234907Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2235249Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2235370Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2235743Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2235930Z getattr(self, test_name)() 2022-11-23T03:54:46.2236298Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2236382Z fn() 2022-11-23T03:54:46.2236752Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2236862Z test(self, **param_kwargs) 2022-11-23T03:54:46.2237221Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2237333Z return func(*args, **kwargs) 2022-11-23T03:54:46.2237593Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2237694Z self.run_subtests( 2022-11-23T03:54:46.2238092Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2238247Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2238617Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2238748Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2239126Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2239233Z output = model(*input) 2022-11-23T03:54:46.2239562Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2239689Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2240068Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2240229Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2240607Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2240714Z _lazy_init(state, module) 2022-11-23T03:54:46.2241068Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2241198Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2241538Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2241648Z return func(*args, **kwargs) 2022-11-23T03:54:46.2242028Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2242118Z p_assert( 2022-11-23T03:54:46.2242454Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2242564Z traceback.print_stack() 2022-11-23T03:54:46.2242785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T03:54:46.2242895Z File "", line 1, in 2022-11-23T03:54:46.2243091Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2243220Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2243404Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2243542Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2243742Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2243833Z self.run() 2022-11-23T03:54:46.2244019Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2244152Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2244494Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2244679Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2245049Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2245159Z getattr(self, test_name)() 2022-11-23T03:54:46.2245528Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2245614Z fn() 2022-11-23T03:54:46.2245984Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2246094Z test(self, **param_kwargs) 2022-11-23T03:54:46.2246447Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2246558Z return func(*args, **kwargs) 2022-11-23T03:54:46.2246817Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2246973Z self.run_subtests( 2022-11-23T03:54:46.2247335Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2247482Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2248128Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2248441Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2249017Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2249122Z output = model(*input) 2022-11-23T03:54:46.2249455Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2249581Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2249963Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2250131Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2250499Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2250606Z _lazy_init(state, module) 2022-11-23T03:54:46.2250960Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2251089Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2251430Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2251534Z return func(*args, **kwargs) 2022-11-23T03:54:46.2251914Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2252003Z p_assert( 2022-11-23T03:54:46.2252338Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2252451Z traceback.print_stack() 2022-11-23T03:54:46.2252672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T03:54:46.2253062Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T03:54:46.2253452Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T03:54:46.2253568Z File "", line 1, in 2022-11-23T03:54:46.2253762Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2253894Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2254081Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2254217Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2254419Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2254594Z self.run() 2022-11-23T03:54:46.2254782Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2254914Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2255267Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2255379Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2255744Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2255854Z getattr(self, test_name)() 2022-11-23T03:54:46.2256218Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2256303Z fn() 2022-11-23T03:54:46.2256671Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2256839Z test(self, **param_kwargs) 2022-11-23T03:54:46.2257206Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2257317Z return func(*args, **kwargs) 2022-11-23T03:54:46.2257575Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2257675Z self.run_subtests( 2022-11-23T03:54:46.2258033Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2258181Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2258547Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2258684Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2259068Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2259177Z output = model(*input) 2022-11-23T03:54:46.2259506Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2259626Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2260005Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2260168Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2260538Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2260647Z _lazy_init(state, module) 2022-11-23T03:54:46.2260999Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2261127Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2261476Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2261587Z return func(*args, **kwargs) 2022-11-23T03:54:46.2261966Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2262054Z p_assert( 2022-11-23T03:54:46.2262389Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2262499Z traceback.print_stack() 2022-11-23T03:54:46.2262720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T03:54:46.2262837Z File "", line 1, in 2022-11-23T03:54:46.2263030Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2263163Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2263350Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2263547Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2263746Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2263836Z self.run() 2022-11-23T03:54:46.2264025Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2264155Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2264501Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2264620Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2264985Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2265095Z getattr(self, test_name)() 2022-11-23T03:54:46.2265459Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2265549Z fn() 2022-11-23T03:54:46.2265976Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2266090Z test(self, **param_kwargs) 2022-11-23T03:54:46.2266455Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2266566Z return func(*args, **kwargs) 2022-11-23T03:54:46.2266822Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2266921Z self.run_subtests( 2022-11-23T03:54:46.2267267Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2267414Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2267780Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2267925Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2268305Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2268410Z output = model(*input) 2022-11-23T03:54:46.2268738Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2268865Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2269242Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2269405Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2269774Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2269882Z _lazy_init(state, module) 2022-11-23T03:54:46.2270239Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2270370Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2270711Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2270824Z return func(*args, **kwargs) 2022-11-23T03:54:46.2271203Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2271291Z p_assert( 2022-11-23T03:54:46.2271626Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2271728Z traceback.print_stack() 2022-11-23T03:54:46.2271946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T03:54:46.2272337Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T03:54:46.2272791Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T03:54:46.2272908Z File "", line 1, in 2022-11-23T03:54:46.2273103Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2273232Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2273419Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2273555Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2273753Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2273842Z self.run() 2022-11-23T03:54:46.2274031Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2274162Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2274506Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2274674Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2275045Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2275156Z getattr(self, test_name)() 2022-11-23T03:54:46.2275505Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2275591Z fn() 2022-11-23T03:54:46.2275959Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2276069Z test(self, **param_kwargs) 2022-11-23T03:54:46.2276431Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2276543Z return func(*args, **kwargs) 2022-11-23T03:54:46.2276800Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2276906Z self.run_subtests( 2022-11-23T03:54:46.2277261Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2277409Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2277774Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2277913Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2278291Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2278395Z output = model(*input) 2022-11-23T03:54:46.2278725Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2278855Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2279238Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2279402Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2279770Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2279870Z _lazy_init(state, module) 2022-11-23T03:54:46.2280222Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2280350Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2280692Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2280803Z return func(*args, **kwargs) 2022-11-23T03:54:46.2281184Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2281273Z p_assert( 2022-11-23T03:54:46.2281612Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2281787Z traceback.print_stack() 2022-11-23T03:54:46.2282005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0 2022-11-23T03:54:46.2282120Z File "", line 1, in 2022-11-23T03:54:46.2282312Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2282440Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2282627Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2282763Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2282960Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2283050Z self.run() 2022-11-23T03:54:46.2283230Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2283361Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2283751Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2283873Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2284240Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2284353Z getattr(self, test_name)() 2022-11-23T03:54:46.2284713Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2284798Z fn() 2022-11-23T03:54:46.2285164Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2285275Z test(self, **param_kwargs) 2022-11-23T03:54:46.2285635Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2285745Z return func(*args, **kwargs) 2022-11-23T03:54:46.2286010Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2286109Z self.run_subtests( 2022-11-23T03:54:46.2286464Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2300563Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2301049Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2301190Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2301574Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2301680Z output = model(*input) 2022-11-23T03:54:46.2302011Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2302152Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2302537Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2302699Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2303070Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2303185Z _lazy_init(state, module) 2022-11-23T03:54:46.2303543Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2303673Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2304011Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2304127Z return func(*args, **kwargs) 2022-11-23T03:54:46.2304511Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2304754Z p_assert( 2022-11-23T03:54:46.2305099Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2305212Z traceback.print_stack() 2022-11-23T03:54:46.2305435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1 2022-11-23T03:54:46.2305831Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 2 nodes. 2022-11-23T03:54:46.2306219Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 2 nodes. 2022-11-23T03:54:46.2306341Z File "", line 1, in 2022-11-23T03:54:46.2306534Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2306667Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2306919Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2307065Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2307269Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2307360Z self.run() 2022-11-23T03:54:46.2307551Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2307674Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2308026Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2308150Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2308518Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2308628Z getattr(self, test_name)() 2022-11-23T03:54:46.2308992Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2309084Z fn() 2022-11-23T03:54:46.2309454Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2309566Z test(self, **param_kwargs) 2022-11-23T03:54:46.2309934Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2310050Z return func(*args, **kwargs) 2022-11-23T03:54:46.2310309Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2310411Z self.run_subtests( 2022-11-23T03:54:46.2310764Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2310915Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2311285Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2311434Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2311813Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2311921Z output = model(*input) 2022-11-23T03:54:46.2312240Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2312372Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2312752Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2312922Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2313298Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2313413Z _lazy_init(state, module) 2022-11-23T03:54:46.2313770Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2313965Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2314311Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2314425Z return func(*args, **kwargs) 2022-11-23T03:54:46.2314810Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2314905Z p_assert( 2022-11-23T03:54:46.2315242Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2315361Z traceback.print_stack() 2022-11-23T03:54:46.2315584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0 2022-11-23T03:54:46.2315705Z File "", line 1, in 2022-11-23T03:54:46.2315904Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2316089Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2316271Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2316411Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2316613Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2316707Z self.run() 2022-11-23T03:54:46.2316901Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2317040Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2317391Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2317516Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2317885Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2318000Z getattr(self, test_name)() 2022-11-23T03:54:46.2318375Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2318465Z fn() 2022-11-23T03:54:46.2318837Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2318954Z test(self, **param_kwargs) 2022-11-23T03:54:46.2319317Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2319433Z return func(*args, **kwargs) 2022-11-23T03:54:46.2319694Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2319785Z self.run_subtests( 2022-11-23T03:54:46.2320141Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2320296Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2320677Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2320822Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2321209Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2321318Z output = model(*input) 2022-11-23T03:54:46.2321653Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2321785Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2322168Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2322335Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2322709Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2322878Z _lazy_init(state, module) 2022-11-23T03:54:46.2323236Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2323370Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2323713Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2323832Z return func(*args, **kwargs) 2022-11-23T03:54:46.2324216Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2324298Z p_assert( 2022-11-23T03:54:46.2324639Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2324756Z traceback.print_stack() 2022-11-23T03:54:46.2324982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1 2022-11-23T03:54:46.2325427Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 2 nodes. 2022-11-23T03:54:46.2325827Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 2 nodes. 2022-11-23T03:54:46.2325952Z File "", line 1, in 2022-11-23T03:54:46.2326151Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2326284Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2326476Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2326618Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2326821Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2326915Z self.run() 2022-11-23T03:54:46.2327108Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2327251Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2327597Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2327941Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2328733Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2328903Z getattr(self, test_name)() 2022-11-23T03:54:46.2329274Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2329364Z fn() 2022-11-23T03:54:46.2329735Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2329854Z test(self, **param_kwargs) 2022-11-23T03:54:46.2330214Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2330338Z return func(*args, **kwargs) 2022-11-23T03:54:46.2330598Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2330702Z self.run_subtests( 2022-11-23T03:54:46.2331061Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2331211Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2331581Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2331724Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2332106Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2332214Z output = model(*input) 2022-11-23T03:54:46.2332549Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2332769Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2333157Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2333321Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2333681Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2333791Z _lazy_init(state, module) 2022-11-23T03:54:46.2334145Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2334276Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2334618Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2334733Z return func(*args, **kwargs) 2022-11-23T03:54:46.2335167Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2335266Z p_assert( 2022-11-23T03:54:46.2335610Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2335725Z traceback.print_stack() 2022-11-23T03:54:46.2335945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1 2022-11-23T03:54:46.2336063Z File "", line 1, in 2022-11-23T03:54:46.2336261Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2336393Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2336584Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2336723Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2336924Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2337009Z self.run() 2022-11-23T03:54:46.2337205Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2337337Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2337679Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2337802Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2338169Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2338283Z getattr(self, test_name)() 2022-11-23T03:54:46.2338646Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2338739Z fn() 2022-11-23T03:54:46.2339111Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2339224Z test(self, **param_kwargs) 2022-11-23T03:54:46.2339594Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2339709Z return func(*args, **kwargs) 2022-11-23T03:54:46.2339969Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2340073Z self.run_subtests( 2022-11-23T03:54:46.2340433Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2340588Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2340945Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2341085Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2341464Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2341669Z output = model(*input) 2022-11-23T03:54:46.2342003Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2342134Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2342517Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2342679Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2343051Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2343167Z _lazy_init(state, module) 2022-11-23T03:54:46.2343522Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2343659Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2344003Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2344190Z return func(*args, **kwargs) 2022-11-23T03:54:46.2344585Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2344678Z p_assert( 2022-11-23T03:54:46.2345015Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2345134Z traceback.print_stack() 2022-11-23T03:54:46.2345356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0 2022-11-23T03:54:46.2345742Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 2 nodes. 2022-11-23T03:54:46.2346133Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 2 nodes. 2022-11-23T03:54:46.2346253Z File "", line 1, in 2022-11-23T03:54:46.2346454Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2346589Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2346781Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2346923Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2347124Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2347218Z self.run() 2022-11-23T03:54:46.2347411Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2347543Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2347888Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2348012Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2348379Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2348503Z getattr(self, test_name)() 2022-11-23T03:54:46.2348867Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2348957Z fn() 2022-11-23T03:54:46.2349317Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2349432Z test(self, **param_kwargs) 2022-11-23T03:54:46.2349793Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2349907Z return func(*args, **kwargs) 2022-11-23T03:54:46.2350168Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2350272Z self.run_subtests( 2022-11-23T03:54:46.2350630Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2350790Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2351221Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2351357Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2351736Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2351841Z output = model(*input) 2022-11-23T03:54:46.2352174Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2352301Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2352686Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2352855Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2353282Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2353404Z _lazy_init(state, module) 2022-11-23T03:54:46.2353760Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2353880Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2354224Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2354342Z return func(*args, **kwargs) 2022-11-23T03:54:46.2354719Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2354812Z p_assert( 2022-11-23T03:54:46.2355148Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2355262Z traceback.print_stack() 2022-11-23T03:54:46.2355483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 1 2022-11-23T03:54:46.2355883Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:8 with 2 nodes. 2022-11-23T03:54:46.2356000Z File "", line 1, in 2022-11-23T03:54:46.2356197Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2356330Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2356521Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2356659Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2356857Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2356951Z self.run() 2022-11-23T03:54:46.2357145Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2357267Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2357617Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2357743Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2358108Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2358221Z getattr(self, test_name)() 2022-11-23T03:54:46.2358582Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2358672Z fn() 2022-11-23T03:54:46.2359038Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2359152Z test(self, **param_kwargs) 2022-11-23T03:54:46.2359515Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2359630Z return func(*args, **kwargs) 2022-11-23T03:54:46.2359890Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2360048Z self.run_subtests( 2022-11-23T03:54:46.2360406Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2360554Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2360921Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2361065Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2361445Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2361554Z output = model(*input) 2022-11-23T03:54:46.2361875Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2362005Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2362435Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2362609Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2362985Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2363095Z _lazy_init(state, module) 2022-11-23T03:54:46.2363450Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2363578Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2363922Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2364036Z return func(*args, **kwargs) 2022-11-23T03:54:46.2364421Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2364510Z p_assert( 2022-11-23T03:54:46.2364853Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2364968Z traceback.print_stack() 2022-11-23T03:54:46.2365192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 0 2022-11-23T03:54:46.2365589Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:8 with 2 nodes. 2022-11-23T03:54:46.2365706Z File "", line 1, in 2022-11-23T03:54:46.2365906Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2366026Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2366215Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2366353Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2366551Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2366651Z self.run() 2022-11-23T03:54:46.2366839Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2366973Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2367316Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2367441Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2367913Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2368061Z getattr(self, test_name)() 2022-11-23T03:54:46.2368428Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2368518Z fn() 2022-11-23T03:54:46.2368882Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2368994Z test(self, **param_kwargs) 2022-11-23T03:54:46.2369436Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2369550Z return func(*args, **kwargs) 2022-11-23T03:54:46.2369801Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2369901Z self.run_subtests( 2022-11-23T03:54:46.2370259Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2370406Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2370772Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2370913Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2371292Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2371456Z output = model(*input) 2022-11-23T03:54:46.2371795Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2371928Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2372307Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2372470Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2372840Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2372952Z _lazy_init(state, module) 2022-11-23T03:54:46.2373306Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2373440Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2373785Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2373905Z return func(*args, **kwargs) 2022-11-23T03:54:46.2374287Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2374367Z p_assert( 2022-11-23T03:54:46.2374706Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2374820Z traceback.print_stack() 2022-11-23T03:54:46.2375041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 1 2022-11-23T03:54:46.2375162Z File "", line 1, in 2022-11-23T03:54:46.2375360Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2375491Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2375681Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2375821Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2376027Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2376121Z self.run() 2022-11-23T03:54:46.2376314Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2376449Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2376794Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2376919Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2377288Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2377389Z getattr(self, test_name)() 2022-11-23T03:54:46.2377749Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2377837Z fn() 2022-11-23T03:54:46.2378207Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2378377Z test(self, **param_kwargs) 2022-11-23T03:54:46.2378743Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2378857Z return func(*args, **kwargs) 2022-11-23T03:54:46.2379115Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2379219Z self.run_subtests( 2022-11-23T03:54:46.2379572Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2379724Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2380093Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2380234Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2380661Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2380770Z output = model(*input) 2022-11-23T03:54:46.2381106Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2381238Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2381619Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2381784Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2382145Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2382256Z _lazy_init(state, module) 2022-11-23T03:54:46.2382610Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2382755Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2383096Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2383212Z return func(*args, **kwargs) 2022-11-23T03:54:46.2383599Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2383690Z p_assert( 2022-11-23T03:54:46.2384026Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2384141Z traceback.print_stack() 2022-11-23T03:54:46.2384365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 0 2022-11-23T03:54:46.2384756Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:9 with 2 nodes. 2022-11-23T03:54:46.2385148Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:9 with 2 nodes. 2022-11-23T03:54:46.2385269Z File "", line 1, in 2022-11-23T03:54:46.2385468Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2385599Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2385787Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2385925Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2386113Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2386209Z self.run() 2022-11-23T03:54:46.2386397Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2386532Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2386877Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2387000Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2387431Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2387547Z getattr(self, test_name)() 2022-11-23T03:54:46.2387909Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2388000Z fn() 2022-11-23T03:54:46.2388368Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2388482Z test(self, **param_kwargs) 2022-11-23T03:54:46.2388846Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2388960Z return func(*args, **kwargs) 2022-11-23T03:54:46.2389217Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2389320Z self.run_subtests( 2022-11-23T03:54:46.2389724Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2389880Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2390240Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2390382Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2390760Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2390872Z output = model(*input) 2022-11-23T03:54:46.2391207Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2391340Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2391722Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2391894Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2392265Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2392376Z _lazy_init(state, module) 2022-11-23T03:54:46.2392731Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2392865Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2393207Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2393321Z return func(*args, **kwargs) 2022-11-23T03:54:46.2393703Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2393794Z p_assert( 2022-11-23T03:54:46.2394133Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2394257Z traceback.print_stack() 2022-11-23T03:54:46.2394471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 0 2022-11-23T03:54:46.2394591Z File "", line 1, in 2022-11-23T03:54:46.2394792Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2394924Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2395115Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2395255Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2395456Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2395549Z self.run() 2022-11-23T03:54:46.2395743Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2395877Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2396222Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2396396Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2396770Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2396883Z getattr(self, test_name)() 2022-11-23T03:54:46.2397248Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2397337Z fn() 2022-11-23T03:54:46.2397707Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2397809Z test(self, **param_kwargs) 2022-11-23T03:54:46.2398172Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2398286Z return func(*args, **kwargs) 2022-11-23T03:54:46.2398591Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2398699Z self.run_subtests( 2022-11-23T03:54:46.2399055Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2399208Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2399578Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2399720Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2400099Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2400211Z output = model(*input) 2022-11-23T03:54:46.2400540Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2400673Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2401059Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2401223Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2401594Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2401704Z _lazy_init(state, module) 2022-11-23T03:54:46.2402062Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2402194Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2402524Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2402637Z return func(*args, **kwargs) 2022-11-23T03:54:46.2403022Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2403119Z p_assert( 2022-11-23T03:54:46.2403459Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2403574Z traceback.print_stack() 2022-11-23T03:54:46.2403800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 1 2022-11-23T03:54:46.2404195Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:10 with 2 nodes. 2022-11-23T03:54:46.2404590Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:10 with 2 nodes. 2022-11-23T03:54:46.2404710Z File "", line 1, in 2022-11-23T03:54:46.2404912Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2405045Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2405236Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2405437Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2405637Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2405730Z self.run() 2022-11-23T03:54:46.2405918Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2406039Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2406384Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2406508Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2406877Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2406989Z getattr(self, test_name)() 2022-11-23T03:54:46.2407350Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2407439Z fn() 2022-11-23T03:54:46.2407934Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2408055Z test(self, **param_kwargs) 2022-11-23T03:54:46.2408425Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2408535Z return func(*args, **kwargs) 2022-11-23T03:54:46.2408793Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2408895Z self.run_subtests( 2022-11-23T03:54:46.2409248Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2409397Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2409763Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2409905Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2410286Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2410392Z output = model(*input) 2022-11-23T03:54:46.2410711Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2410844Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2411224Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2411383Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2411756Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2411868Z _lazy_init(state, module) 2022-11-23T03:54:46.2412221Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2412361Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2412702Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2412813Z return func(*args, **kwargs) 2022-11-23T03:54:46.2413195Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2413287Z p_assert( 2022-11-23T03:54:46.2413620Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2413734Z traceback.print_stack() 2022-11-23T03:54:46.2413958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 0 2022-11-23T03:54:46.2414074Z File "", line 1, in 2022-11-23T03:54:46.2414271Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2414389Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2414659Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2414796Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2414996Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2415085Z self.run() 2022-11-23T03:54:46.2415277Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2415409Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2415755Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2415879Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2416250Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2416360Z getattr(self, test_name)() 2022-11-23T03:54:46.2416768Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2416861Z fn() 2022-11-23T03:54:46.2417230Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2417343Z test(self, **param_kwargs) 2022-11-23T03:54:46.2417704Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2417815Z return func(*args, **kwargs) 2022-11-23T03:54:46.2418066Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2418170Z self.run_subtests( 2022-11-23T03:54:46.2418530Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2418679Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2419052Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2419192Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2419574Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2419685Z output = model(*input) 2022-11-23T03:54:46.2420011Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2420140Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2420516Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2420678Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2421047Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2421156Z _lazy_init(state, module) 2022-11-23T03:54:46.2421515Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2421643Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2421980Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2422089Z return func(*args, **kwargs) 2022-11-23T03:54:46.2422467Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2422547Z p_assert( 2022-11-23T03:54:46.2422880Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2422989Z traceback.print_stack() 2022-11-23T03:54:46.2423210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 1 2022-11-23T03:54:46.2423603Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:11 with 2 nodes. 2022-11-23T03:54:46.2424055Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:11 with 2 nodes. 2022-11-23T03:54:46.2424170Z File "", line 1, in 2022-11-23T03:54:46.2424364Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2424490Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2424677Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2424813Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2425010Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2425099Z self.run() 2022-11-23T03:54:46.2425287Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2425417Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2425826Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2425948Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2426319Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2426421Z getattr(self, test_name)() 2022-11-23T03:54:46.2426781Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2426866Z fn() 2022-11-23T03:54:46.2427231Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2427344Z test(self, **param_kwargs) 2022-11-23T03:54:46.2427704Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2427814Z return func(*args, **kwargs) 2022-11-23T03:54:46.2428075Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2428180Z self.run_subtests( 2022-11-23T03:54:46.2428532Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2428678Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2429041Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2429184Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2429560Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2429664Z output = model(*input) 2022-11-23T03:54:46.2429990Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2430117Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2430502Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2430655Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2431023Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2431130Z _lazy_init(state, module) 2022-11-23T03:54:46.2431479Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2431607Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2431947Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2432058Z return func(*args, **kwargs) 2022-11-23T03:54:46.2432442Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2432587Z p_assert( 2022-11-23T03:54:46.2432928Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2433043Z traceback.print_stack() 2022-11-23T03:54:46.2433264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 1 2022-11-23T03:54:46.2433384Z File "", line 1, in 2022-11-23T03:54:46.2433581Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2433712Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2433901Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2434042Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2434241Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2434322Z self.run() 2022-11-23T03:54:46.2434563Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2434703Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2435046Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2435167Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2435531Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2435644Z getattr(self, test_name)() 2022-11-23T03:54:46.2436006Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2436095Z fn() 2022-11-23T03:54:46.2436468Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2436581Z test(self, **param_kwargs) 2022-11-23T03:54:46.2436947Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2437065Z return func(*args, **kwargs) 2022-11-23T03:54:46.2437324Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2437426Z self.run_subtests( 2022-11-23T03:54:46.2437781Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2437934Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2438291Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2438437Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2438818Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2438933Z output = model(*input) 2022-11-23T03:54:46.2439271Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2439403Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2439789Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2439954Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2440326Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2440439Z _lazy_init(state, module) 2022-11-23T03:54:46.2440793Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2440925Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2441265Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2441382Z return func(*args, **kwargs) 2022-11-23T03:54:46.2441822Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2441917Z p_assert( 2022-11-23T03:54:46.2442253Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2442371Z traceback.print_stack() 2022-11-23T03:54:46.2442582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 0 2022-11-23T03:54:46.2442976Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:12 with 2 nodes. 2022-11-23T03:54:46.2443368Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:12 with 2 nodes. 2022-11-23T03:54:46.2443484Z File "", line 1, in 2022-11-23T03:54:46.2443679Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2443859Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2444047Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2444182Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2444380Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2444468Z self.run() 2022-11-23T03:54:46.2444657Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2444790Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2445129Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2445252Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2445618Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2445731Z getattr(self, test_name)() 2022-11-23T03:54:46.2446103Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2446197Z fn() 2022-11-23T03:54:46.2446555Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2446668Z test(self, **param_kwargs) 2022-11-23T03:54:46.2447030Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2447144Z return func(*args, **kwargs) 2022-11-23T03:54:46.2447406Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2447512Z self.run_subtests( 2022-11-23T03:54:46.2448080Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2448395Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2448861Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2449004Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2449384Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2449495Z output = model(*input) 2022-11-23T03:54:46.2449826Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2449959Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2450342Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2450508Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2450882Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2451079Z _lazy_init(state, module) 2022-11-23T03:54:46.2451440Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2451559Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2451900Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2452016Z return func(*args, **kwargs) 2022-11-23T03:54:46.2452397Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2452495Z p_assert( 2022-11-23T03:54:46.2452834Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2452948Z traceback.print_stack() 2022-11-23T03:54:46.2453168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 0 2022-11-23T03:54:46.2453288Z File "", line 1, in 2022-11-23T03:54:46.2453551Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2453689Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2453880Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2454024Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2454227Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2454322Z self.run() 2022-11-23T03:54:46.2454515Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2454650Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2454986Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2455106Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2455477Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2455596Z getattr(self, test_name)() 2022-11-23T03:54:46.2455962Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2456049Z fn() 2022-11-23T03:54:46.2456422Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2456540Z test(self, **param_kwargs) 2022-11-23T03:54:46.2456902Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2457014Z return func(*args, **kwargs) 2022-11-23T03:54:46.2457272Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2457379Z self.run_subtests( 2022-11-23T03:54:46.2457740Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2457892Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2458259Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2458403Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2458785Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2458893Z output = model(*input) 2022-11-23T03:54:46.2459211Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2459344Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2459725Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2459891Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2460324Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2460436Z _lazy_init(state, module) 2022-11-23T03:54:46.2460794Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2460929Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2461272Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2461393Z return func(*args, **kwargs) 2022-11-23T03:54:46.2461774Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2461868Z p_assert( 2022-11-23T03:54:46.2462205Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2462318Z traceback.print_stack() 2022-11-23T03:54:46.2462592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 1 2022-11-23T03:54:46.2462994Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:13 with 2 nodes. 2022-11-23T03:54:46.2463385Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:13 with 2 nodes. 2022-11-23T03:54:46.2463505Z File "", line 1, in 2022-11-23T03:54:46.2463706Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2463825Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2464018Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2464158Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2464360Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2464453Z self.run() 2022-11-23T03:54:46.2464652Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2464786Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2465130Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2465253Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2465621Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2465734Z getattr(self, test_name)() 2022-11-23T03:54:46.2466100Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2466187Z fn() 2022-11-23T03:54:46.2466558Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2466675Z test(self, **param_kwargs) 2022-11-23T03:54:46.2467040Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2467160Z return func(*args, **kwargs) 2022-11-23T03:54:46.2467409Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2467514Z self.run_subtests( 2022-11-23T03:54:46.2467869Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2468019Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2468389Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2468531Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2468909Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2469019Z output = model(*input) 2022-11-23T03:54:46.2469410Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2469542Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2469926Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2470093Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2470463Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2470576Z _lazy_init(state, module) 2022-11-23T03:54:46.2470930Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2471061Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2471401Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2471560Z return func(*args, **kwargs) 2022-11-23T03:54:46.2471950Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2472030Z p_assert( 2022-11-23T03:54:46.2472367Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2472482Z traceback.print_stack() 2022-11-23T03:54:46.2472711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 1 2022-11-23T03:54:46.2472829Z File "", line 1, in 2022-11-23T03:54:46.2473028Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2473161Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2473355Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2473501Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2473709Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2473805Z self.run() 2022-11-23T03:54:46.2473997Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2474130Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2474475Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2474596Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2474962Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2475064Z getattr(self, test_name)() 2022-11-23T03:54:46.2475426Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2475515Z fn() 2022-11-23T03:54:46.2475884Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2476006Z test(self, **param_kwargs) 2022-11-23T03:54:46.2476366Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2476481Z return func(*args, **kwargs) 2022-11-23T03:54:46.2476740Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2476844Z self.run_subtests( 2022-11-23T03:54:46.2477197Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2477352Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2477721Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2477863Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2478246Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2478405Z output = model(*input) 2022-11-23T03:54:46.2478740Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2478872Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2479252Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2479417Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2479777Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2479890Z _lazy_init(state, module) 2022-11-23T03:54:46.2480245Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2480373Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2480778Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2480895Z return func(*args, **kwargs) 2022-11-23T03:54:46.2481278Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2481373Z p_assert( 2022-11-23T03:54:46.2481716Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2481830Z traceback.print_stack() 2022-11-23T03:54:46.2482053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 0 2022-11-23T03:54:46.2482445Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:14 with 2 nodes. 2022-11-23T03:54:46.2482835Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:14 with 2 nodes. 2022-11-23T03:54:46.2483063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 1 2022-11-23T03:54:46.2483451Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:15 with 2 nodes. 2022-11-23T03:54:46.2483674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 0 2022-11-23T03:54:46.2484068Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:15 with 2 nodes. 2022-11-23T03:54:46.2484289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 0 2022-11-23T03:54:46.2484511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 1 2022-11-23T03:54:46.2484903Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:16 with 2 nodes. 2022-11-23T03:54:46.2485297Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:16 with 2 nodes. 2022-11-23T03:54:46.2486073Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2486300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 0 2022-11-23T03:54:46.2486691Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:17 with 2 nodes. 2022-11-23T03:54:46.2486899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 1 2022-11-23T03:54:46.2487288Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:17 with 2 nodes. 2022-11-23T03:54:46.2487565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 1 2022-11-23T03:54:46.2487844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 0 2022-11-23T03:54:46.2488248Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:18 with 2 nodes. 2022-11-23T03:54:46.2488637Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:18 with 2 nodes. 2022-11-23T03:54:46.2489406Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2489625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 0 2022-11-23T03:54:46.2490018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 1 2022-11-23T03:54:46.2490419Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:19 with 2 nodes. 2022-11-23T03:54:46.2490807Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:19 with 2 nodes. 2022-11-23T03:54:46.2491579Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2492336Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2492566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:20 to store for rank: 1 2022-11-23T03:54:46.2492788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:20 to store for rank: 0 2022-11-23T03:54:46.2493180Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:20 with 2 nodes. 2022-11-23T03:54:46.2493564Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:20 with 2 nodes. 2022-11-23T03:54:46.2494326Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2494560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:21 to store for rank: 1 2022-11-23T03:54:46.2494782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:21 to store for rank: 0 2022-11-23T03:54:46.2495173Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:21 with 2 nodes. 2022-11-23T03:54:46.2495564Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:21 with 2 nodes. 2022-11-23T03:54:46.2495785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:22 to store for rank: 1 2022-11-23T03:54:46.2496007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:22 to store for rank: 0 2022-11-23T03:54:46.2496397Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:22 with 2 nodes. 2022-11-23T03:54:46.2496793Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:22 with 2 nodes. 2022-11-23T03:54:46.2497610Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2497834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:23 to store for rank: 0 2022-11-23T03:54:46.2498053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:23 to store for rank: 1 2022-11-23T03:54:46.2498444Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:23 with 2 nodes. 2022-11-23T03:54:46.2498831Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:23 with 2 nodes. 2022-11-23T03:54:46.2499101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:24 to store for rank: 1 2022-11-23T03:54:46.2499330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:24 to store for rank: 0 2022-11-23T03:54:46.2499724Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:24 with 2 nodes. 2022-11-23T03:54:46.2500108Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:24 with 2 nodes. 2022-11-23T03:54:46.2500872Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2501095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:25 to store for rank: 1 2022-11-23T03:54:46.2501319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:25 to store for rank: 0 2022-11-23T03:54:46.2501708Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:25 with 2 nodes. 2022-11-23T03:54:46.2502099Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:25 with 2 nodes. 2022-11-23T03:54:46.2502321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:26 to store for rank: 1 2022-11-23T03:54:46.2502710Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:26 with 2 nodes. 2022-11-23T03:54:46.2502933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:26 to store for rank: 0 2022-11-23T03:54:46.2503323Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:26 with 2 nodes. 2022-11-23T03:54:46.2504093Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2504321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:27 to store for rank: 1 2022-11-23T03:54:46.2504531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:27 to store for rank: 0 2022-11-23T03:54:46.2504927Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:27 with 2 nodes. 2022-11-23T03:54:46.2505320Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:27 with 2 nodes. 2022-11-23T03:54:46.2505540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:28 to store for rank: 1 2022-11-23T03:54:46.2505813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:28 to store for rank: 0 2022-11-23T03:54:46.2506209Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:28 with 2 nodes. 2022-11-23T03:54:46.2506594Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:28 with 2 nodes. 2022-11-23T03:54:46.2507357Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2507580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:29 to store for rank: 1 2022-11-23T03:54:46.2507801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:29 to store for rank: 0 2022-11-23T03:54:46.2508238Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:29 with 2 nodes. 2022-11-23T03:54:46.2508636Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:29 with 2 nodes. 2022-11-23T03:54:46.2508857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:30 to store for rank: 1 2022-11-23T03:54:46.2509076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:30 to store for rank: 0 2022-11-23T03:54:46.2509464Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:30 with 2 nodes. 2022-11-23T03:54:46.2509854Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:30 with 2 nodes. 2022-11-23T03:54:46.2510612Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2510839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:31 to store for rank: 1 2022-11-23T03:54:46.2511061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:31 to store for rank: 0 2022-11-23T03:54:46.2511452Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:31 with 2 nodes. 2022-11-23T03:54:46.2511839Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:31 with 2 nodes. 2022-11-23T03:54:46.2512061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:32 to store for rank: 1 2022-11-23T03:54:46.2512278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:32 to store for rank: 0 2022-11-23T03:54:46.2512672Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:32 with 2 nodes. 2022-11-23T03:54:46.2513060Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:32 with 2 nodes. 2022-11-23T03:54:46.2513822Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2514045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:33 to store for rank: 1 2022-11-23T03:54:46.2514268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:33 to store for rank: 0 2022-11-23T03:54:46.2514661Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:33 with 2 nodes. 2022-11-23T03:54:46.2515101Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:33 with 2 nodes. 2022-11-23T03:54:46.2515324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:34 to store for rank: 1 2022-11-23T03:54:46.2515540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:34 to store for rank: 0 2022-11-23T03:54:46.2515929Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:34 with 2 nodes. 2022-11-23T03:54:46.2516316Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:34 with 2 nodes. 2022-11-23T03:54:46.2517117Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2517878Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2518100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:35 to store for rank: 0 2022-11-23T03:54:46.2518320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:35 to store for rank: 1 2022-11-23T03:54:46.2518707Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:35 with 2 nodes. 2022-11-23T03:54:46.2519097Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:35 with 2 nodes. 2022-11-23T03:54:46.2519860Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2520085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:36 to store for rank: 1 2022-11-23T03:54:46.2520305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:36 to store for rank: 0 2022-11-23T03:54:46.2520694Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:36 with 2 nodes. 2022-11-23T03:54:46.2521069Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:36 with 2 nodes. 2022-11-23T03:54:46.2521831Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2522066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:37 to store for rank: 1 2022-11-23T03:54:46.2522272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:37 to store for rank: 0 2022-11-23T03:54:46.2522659Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:37 with 2 nodes. 2022-11-23T03:54:46.2523052Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:37 with 2 nodes. 2022-11-23T03:54:46.2523155Z dist init r=1, world=2 2022-11-23T03:54:46.2523467Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.2523831Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.2524135Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.2524437Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.2524739Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.2525038Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.2525373Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.2525677Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.2525975Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.2526275Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.2526577Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.2526879Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.2526984Z dist init r=0, world=2 2022-11-23T03:54:46.2527291Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.2527587Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.2528314Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.2528899Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.2529201Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.2529509Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.2529807Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.2530101Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.2530397Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.2530697Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.2530994Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.2531431Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.2531524Z ok (32.775s) 2022-11-23T03:54:46.2531873Z test_mixture_of_experts_with_delay_before_free_offload_true_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47841 2022-11-23T03:54:46.2532081Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47842 2022-11-23T03:54:46.2532495Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.2532660Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.2533109Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.2533294Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.2533516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.2533896Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.2534060Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.2534431Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.2534608Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.2534833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.2535236Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.2535634Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.2535909Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.2536185Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.2536400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.2536616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.2537670Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.2537773Z warnings.warn( 2022-11-23T03:54:46.2537999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T03:54:46.2539033Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.2539134Z warnings.warn( 2022-11-23T03:54:46.2539358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T03:54:46.2539755Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:54:46.2540203Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:54:46.2540326Z File "", line 1, in 2022-11-23T03:54:46.2540531Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2540666Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2540861Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2541004Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2541211Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2541304Z self.run() 2022-11-23T03:54:46.2541495Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2541618Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2542019Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2542152Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2542526Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2542643Z getattr(self, test_name)() 2022-11-23T03:54:46.2543012Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2543102Z fn() 2022-11-23T03:54:46.2543476Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2543592Z test(self, **param_kwargs) 2022-11-23T03:54:46.2543956Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2544069Z return func(*args, **kwargs) 2022-11-23T03:54:46.2544337Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2544442Z self.run_subtests( 2022-11-23T03:54:46.2544806Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2544960Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2545328Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2545472Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2545854Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2545963Z output = model(*input) 2022-11-23T03:54:46.2546284Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2546413Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2546801Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2546971Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2547345Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2547459Z _lazy_init(state, module) 2022-11-23T03:54:46.2547818Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2547950Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2548293Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2548412Z return func(*args, **kwargs) 2022-11-23T03:54:46.2548795Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2548953Z p_assert( 2022-11-23T03:54:46.2549296Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2549416Z traceback.print_stack() 2022-11-23T03:54:46.2549646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T03:54:46.2549764Z File "", line 1, in 2022-11-23T03:54:46.2549965Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2550099Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2550276Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2550416Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2550620Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2550716Z self.run() 2022-11-23T03:54:46.2550950Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2551090Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2551440Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2551563Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2551928Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2552043Z getattr(self, test_name)() 2022-11-23T03:54:46.2552408Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2552497Z fn() 2022-11-23T03:54:46.2552867Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2552983Z test(self, **param_kwargs) 2022-11-23T03:54:46.2553349Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2553466Z return func(*args, **kwargs) 2022-11-23T03:54:46.2553724Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2553816Z self.run_subtests( 2022-11-23T03:54:46.2554179Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2554331Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2554699Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2554845Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2555224Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2555333Z output = model(*input) 2022-11-23T03:54:46.2555670Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2555801Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2556182Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2556344Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2556716Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2556826Z _lazy_init(state, module) 2022-11-23T03:54:46.2557181Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2557310Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2557649Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2557761Z return func(*args, **kwargs) 2022-11-23T03:54:46.2558203Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2558283Z p_assert( 2022-11-23T03:54:46.2558624Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2558736Z traceback.print_stack() 2022-11-23T03:54:46.2558963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T03:54:46.2559358Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T03:54:46.2559746Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T03:54:46.2559867Z File "", line 1, in 2022-11-23T03:54:46.2560063Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2560240Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2560433Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2560574Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2560776Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2560870Z self.run() 2022-11-23T03:54:46.2561058Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2561193Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2561540Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2561670Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2562036Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2562139Z getattr(self, test_name)() 2022-11-23T03:54:46.2562510Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2562599Z fn() 2022-11-23T03:54:46.2562971Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2563087Z test(self, **param_kwargs) 2022-11-23T03:54:46.2563452Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2563567Z return func(*args, **kwargs) 2022-11-23T03:54:46.2563832Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2563937Z self.run_subtests( 2022-11-23T03:54:46.2564296Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2564448Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2564826Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2564966Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2565344Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2565452Z output = model(*input) 2022-11-23T03:54:46.2565785Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2565918Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2566300Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2566452Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2566830Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2567002Z _lazy_init(state, module) 2022-11-23T03:54:46.2567362Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2567493Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2567930Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2568049Z return func(*args, **kwargs) 2022-11-23T03:54:46.2568435Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2568528Z p_assert( 2022-11-23T03:54:46.2568866Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2568982Z traceback.print_stack() 2022-11-23T03:54:46.2569206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T03:54:46.2569327Z File "", line 1, in 2022-11-23T03:54:46.2569591Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2569730Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2569921Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2570063Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2570266Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2570348Z self.run() 2022-11-23T03:54:46.2570537Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2570670Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2571021Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2571146Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2571518Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2571637Z getattr(self, test_name)() 2022-11-23T03:54:46.2571999Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2572089Z fn() 2022-11-23T03:54:46.2572459Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2572574Z test(self, **param_kwargs) 2022-11-23T03:54:46.2572940Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2573055Z return func(*args, **kwargs) 2022-11-23T03:54:46.2573318Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2573420Z self.run_subtests( 2022-11-23T03:54:46.2573783Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2573939Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2574296Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2574443Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2574824Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2574938Z output = model(*input) 2022-11-23T03:54:46.2575269Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2575404Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2575789Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2575954Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2576395Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2576509Z _lazy_init(state, module) 2022-11-23T03:54:46.2576864Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2576999Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2577342Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2577460Z return func(*args, **kwargs) 2022-11-23T03:54:46.2577842Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2577935Z p_assert( 2022-11-23T03:54:46.2578274Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2578390Z traceback.print_stack() 2022-11-23T03:54:46.2578683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T03:54:46.2579073Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T03:54:46.2579463Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T03:54:46.2579585Z File "", line 1, in 2022-11-23T03:54:46.2579786Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2579921Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2580112Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2580251Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2580454Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2580550Z self.run() 2022-11-23T03:54:46.2580747Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2580884Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2581228Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2581352Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2581720Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2581830Z getattr(self, test_name)() 2022-11-23T03:54:46.2582197Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2582291Z fn() 2022-11-23T03:54:46.2582650Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2582769Z test(self, **param_kwargs) 2022-11-23T03:54:46.2583132Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2583250Z return func(*args, **kwargs) 2022-11-23T03:54:46.2583512Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2583617Z self.run_subtests( 2022-11-23T03:54:46.2583978Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2584135Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2584502Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2584642Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2585022Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2585133Z output = model(*input) 2022-11-23T03:54:46.2585526Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2585656Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2586036Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2586203Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2586577Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2586690Z _lazy_init(state, module) 2022-11-23T03:54:46.2587052Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2587171Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2587513Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2587681Z return func(*args, **kwargs) 2022-11-23T03:54:46.2588070Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2588162Z p_assert( 2022-11-23T03:54:46.2588502Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2588620Z traceback.print_stack() 2022-11-23T03:54:46.2588846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1 2022-11-23T03:54:46.2588968Z File "", line 1, in 2022-11-23T03:54:46.2589164Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2589295Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2589486Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2589628Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2589839Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2589932Z self.run() 2022-11-23T03:54:46.2590125Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2590258Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2590590Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2590712Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2591081Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2591198Z getattr(self, test_name)() 2022-11-23T03:54:46.2591562Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2591654Z fn() 2022-11-23T03:54:46.2592026Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2592142Z test(self, **param_kwargs) 2022-11-23T03:54:46.2592506Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2592625Z return func(*args, **kwargs) 2022-11-23T03:54:46.2592887Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2592990Z self.run_subtests( 2022-11-23T03:54:46.2593348Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2593498Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2593867Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2594012Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2594399Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2594566Z output = model(*input) 2022-11-23T03:54:46.2594888Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2595021Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2595402Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2595566Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2595939Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2596052Z _lazy_init(state, module) 2022-11-23T03:54:46.2596411Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2596544Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2596936Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2597054Z return func(*args, **kwargs) 2022-11-23T03:54:46.2597439Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2597533Z p_assert( 2022-11-23T03:54:46.2597871Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2597985Z traceback.print_stack() 2022-11-23T03:54:46.2598209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0 2022-11-23T03:54:46.2598599Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 2 nodes. 2022-11-23T03:54:46.2598987Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 2 nodes. 2022-11-23T03:54:46.2599114Z File "", line 1, in 2022-11-23T03:54:46.2599318Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2599438Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2599630Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2599775Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2599978Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2600072Z self.run() 2022-11-23T03:54:46.2600264Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2600400Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2600747Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2600871Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2601241Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2601359Z getattr(self, test_name)() 2022-11-23T03:54:46.2601722Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2601812Z fn() 2022-11-23T03:54:46.2602184Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2602300Z test(self, **param_kwargs) 2022-11-23T03:54:46.2602663Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2602776Z return func(*args, **kwargs) 2022-11-23T03:54:46.2603024Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2603129Z self.run_subtests( 2022-11-23T03:54:46.2603489Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2603696Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2604069Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2604220Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2604603Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2604710Z output = model(*input) 2022-11-23T03:54:46.2605043Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2605178Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2605561Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2605728Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2606149Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2606263Z _lazy_init(state, module) 2022-11-23T03:54:46.2606625Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2606758Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2607103Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2607220Z return func(*args, **kwargs) 2022-11-23T03:54:46.2607607Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2607751Z p_assert( 2022-11-23T03:54:46.2608354Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2608475Z traceback.print_stack() 2022-11-23T03:54:46.2608705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0 2022-11-23T03:54:46.2609105Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 2 nodes. 2022-11-23T03:54:46.2609229Z File "", line 1, in 2022-11-23T03:54:46.2609426Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2609557Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2609750Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2609893Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2610095Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2610191Z self.run() 2022-11-23T03:54:46.2610383Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2610519Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2610866Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2610991Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2611359Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2611461Z getattr(self, test_name)() 2022-11-23T03:54:46.2611828Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2611918Z fn() 2022-11-23T03:54:46.2612291Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2612410Z test(self, **param_kwargs) 2022-11-23T03:54:46.2612772Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2612888Z return func(*args, **kwargs) 2022-11-23T03:54:46.2613237Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2613342Z self.run_subtests( 2022-11-23T03:54:46.2613705Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2613857Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2614225Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2614366Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2614815Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2615059Z output = model(*input) 2022-11-23T03:54:46.2615816Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2616219Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2617106Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2617450Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2618304Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2618552Z _lazy_init(state, module) 2022-11-23T03:54:46.2619369Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2619657Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2620433Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2620687Z return func(*args, **kwargs) 2022-11-23T03:54:46.2621578Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2621784Z p_assert( 2022-11-23T03:54:46.2622557Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2622806Z traceback.print_stack() 2022-11-23T03:54:46.2623308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1 2022-11-23T03:54:46.2624204Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 2 nodes. 2022-11-23T03:54:46.2624467Z File "", line 1, in 2022-11-23T03:54:46.2624907Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2625198Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2625623Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2625931Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2626382Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2626584Z self.run() 2022-11-23T03:54:46.2627010Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2627308Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2628093Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2628365Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2629206Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2629457Z getattr(self, test_name)() 2022-11-23T03:54:46.2630284Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2630477Z fn() 2022-11-23T03:54:46.2631333Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2631690Z test(self, **param_kwargs) 2022-11-23T03:54:46.2632529Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2632786Z return func(*args, **kwargs) 2022-11-23T03:54:46.2633365Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2633588Z self.run_subtests( 2022-11-23T03:54:46.2634397Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2634731Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2635561Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2635876Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2636970Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2637217Z output = model(*input) 2022-11-23T03:54:46.2637983Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2638271Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2639148Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2639516Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2640365Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2640611Z _lazy_init(state, module) 2022-11-23T03:54:46.2641416Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2641706Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2642506Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2642760Z return func(*args, **kwargs) 2022-11-23T03:54:46.2643635Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2643832Z p_assert( 2022-11-23T03:54:46.2644600Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2644849Z traceback.print_stack() 2022-11-23T03:54:46.2645329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1 2022-11-23T03:54:46.2645589Z File "", line 1, in 2022-11-23T03:54:46.2646027Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2646314Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2646752Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2647067Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2647513Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2647944Z self.run() 2022-11-23T03:54:46.2648375Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2648668Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2649459Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2649727Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2650569Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2650823Z getattr(self, test_name)() 2022-11-23T03:54:46.2651647Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2651995Z fn() 2022-11-23T03:54:46.2652848Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2653089Z test(self, **param_kwargs) 2022-11-23T03:54:46.2653918Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2654170Z return func(*args, **kwargs) 2022-11-23T03:54:46.2654761Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2654984Z self.run_subtests( 2022-11-23T03:54:46.2655382Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2655537Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2655958Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2656108Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2656493Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2656608Z output = model(*input) 2022-11-23T03:54:46.2656939Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2657070Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2657450Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2657615Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2657988Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2658098Z _lazy_init(state, module) 2022-11-23T03:54:46.2658459Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2658594Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2658924Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2659040Z return func(*args, **kwargs) 2022-11-23T03:54:46.2659423Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2659521Z p_assert( 2022-11-23T03:54:46.2659861Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2659980Z traceback.print_stack() 2022-11-23T03:54:46.2660205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0 2022-11-23T03:54:46.2660598Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 2 nodes. 2022-11-23T03:54:46.2660993Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 2 nodes. 2022-11-23T03:54:46.2661111Z File "", line 1, in 2022-11-23T03:54:46.2661310Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2661447Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2661639Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2661781Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2661987Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2662081Z self.run() 2022-11-23T03:54:46.2662277Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2662399Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2662750Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2662932Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2663306Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2663419Z getattr(self, test_name)() 2022-11-23T03:54:46.2663781Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2663869Z fn() 2022-11-23T03:54:46.2664236Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2664352Z test(self, **param_kwargs) 2022-11-23T03:54:46.2664713Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2664830Z return func(*args, **kwargs) 2022-11-23T03:54:46.2665136Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2665245Z self.run_subtests( 2022-11-23T03:54:46.2665605Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2665755Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2666123Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2666267Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2666653Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2666750Z output = model(*input) 2022-11-23T03:54:46.2667084Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2667219Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2667602Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2667769Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2668145Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2668263Z _lazy_init(state, module) 2022-11-23T03:54:46.2668621Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2668754Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2669097Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2669212Z return func(*args, **kwargs) 2022-11-23T03:54:46.2669593Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2669687Z p_assert( 2022-11-23T03:54:46.2670035Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2670149Z traceback.print_stack() 2022-11-23T03:54:46.2670370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 0 2022-11-23T03:54:46.2670491Z File "", line 1, in 2022-11-23T03:54:46.2670689Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2670806Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2670997Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2671140Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2671346Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2671441Z self.run() 2022-11-23T03:54:46.2671639Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2671844Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2672191Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2672318Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2672683Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2672799Z getattr(self, test_name)() 2022-11-23T03:54:46.2673164Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2673253Z fn() 2022-11-23T03:54:46.2673624Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2673738Z test(self, **param_kwargs) 2022-11-23T03:54:46.2674101Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2674263Z return func(*args, **kwargs) 2022-11-23T03:54:46.2674515Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2674619Z self.run_subtests( 2022-11-23T03:54:46.2674977Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2675126Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2675495Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2675637Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2676018Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2676125Z output = model(*input) 2022-11-23T03:54:46.2676459Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2676598Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2676979Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2677145Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2677521Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2677634Z _lazy_init(state, module) 2022-11-23T03:54:46.2677992Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2678124Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2678468Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2678584Z return func(*args, **kwargs) 2022-11-23T03:54:46.2678972Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2679055Z p_assert( 2022-11-23T03:54:46.2679397Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2679511Z traceback.print_stack() 2022-11-23T03:54:46.2679735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 1 2022-11-23T03:54:46.2680126Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:8 with 2 nodes. 2022-11-23T03:54:46.2680515Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:8 with 2 nodes. 2022-11-23T03:54:46.2680637Z File "", line 1, in 2022-11-23T03:54:46.2680835Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2680965Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2681212Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2681352Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2681556Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2681651Z self.run() 2022-11-23T03:54:46.2681843Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2681981Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2682329Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2682452Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2682822Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2682925Z getattr(self, test_name)() 2022-11-23T03:54:46.2683330Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2683423Z fn() 2022-11-23T03:54:46.2683799Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2683916Z test(self, **param_kwargs) 2022-11-23T03:54:46.2684279Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2684395Z return func(*args, **kwargs) 2022-11-23T03:54:46.2684656Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2684761Z self.run_subtests( 2022-11-23T03:54:46.2685118Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2685276Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2685649Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2685792Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2686173Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2686285Z output = model(*input) 2022-11-23T03:54:46.2686618Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2686749Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2687131Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2687285Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2687661Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2687816Z _lazy_init(state, module) 2022-11-23T03:54:46.2688179Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2688313Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2688657Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2688772Z return func(*args, **kwargs) 2022-11-23T03:54:46.2689158Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2689251Z p_assert( 2022-11-23T03:54:46.2689592Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2689708Z traceback.print_stack() 2022-11-23T03:54:46.2689931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 1 2022-11-23T03:54:46.2690049Z File "", line 1, in 2022-11-23T03:54:46.2690249Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2690447Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2690641Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2690782Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2690983Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2691065Z self.run() 2022-11-23T03:54:46.2691259Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2691392Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2691744Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2691871Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2692237Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2692405Z getattr(self, test_name)() 2022-11-23T03:54:46.2692774Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2692864Z fn() 2022-11-23T03:54:46.2693235Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2693351Z test(self, **param_kwargs) 2022-11-23T03:54:46.2693715Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2693831Z return func(*args, **kwargs) 2022-11-23T03:54:46.2694094Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2694199Z self.run_subtests( 2022-11-23T03:54:46.2694558Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2694717Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2695074Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2695219Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2695599Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2695712Z output = model(*input) 2022-11-23T03:54:46.2696043Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2696176Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2696560Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2696727Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2697106Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2697221Z _lazy_init(state, module) 2022-11-23T03:54:46.2697578Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2697712Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2698055Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2698171Z return func(*args, **kwargs) 2022-11-23T03:54:46.2698552Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2698645Z p_assert( 2022-11-23T03:54:46.2698982Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2699097Z traceback.print_stack() 2022-11-23T03:54:46.2699310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 0 2022-11-23T03:54:46.2699765Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:9 with 2 nodes. 2022-11-23T03:54:46.2700158Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:9 with 2 nodes. 2022-11-23T03:54:46.2700279Z File "", line 1, in 2022-11-23T03:54:46.2700478Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2700610Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2700802Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2700943Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2701143Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2701236Z self.run() 2022-11-23T03:54:46.2701430Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2701611Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2701957Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2702083Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2702458Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2702574Z getattr(self, test_name)() 2022-11-23T03:54:46.2702939Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2703027Z fn() 2022-11-23T03:54:46.2703385Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2703500Z test(self, **param_kwargs) 2022-11-23T03:54:46.2703863Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2703985Z return func(*args, **kwargs) 2022-11-23T03:54:46.2704247Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2704355Z self.run_subtests( 2022-11-23T03:54:46.2704711Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2704864Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2705235Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2705378Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2705764Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2705873Z output = model(*input) 2022-11-23T03:54:46.2706211Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2706346Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2706729Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2706900Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2707273Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2707388Z _lazy_init(state, module) 2022-11-23T03:54:46.2707744Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2707863Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2708204Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2708321Z return func(*args, **kwargs) 2022-11-23T03:54:46.2708770Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2708868Z p_assert( 2022-11-23T03:54:46.2709208Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2709324Z traceback.print_stack() 2022-11-23T03:54:46.2709549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 1 2022-11-23T03:54:46.2709667Z File "", line 1, in 2022-11-23T03:54:46.2709863Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2709996Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2710191Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2710336Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2710539Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2710683Z self.run() 2022-11-23T03:54:46.2710876Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2711010Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2711345Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2711470Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2711839Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2711952Z getattr(self, test_name)() 2022-11-23T03:54:46.2712316Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2712408Z fn() 2022-11-23T03:54:46.2712781Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2712896Z test(self, **param_kwargs) 2022-11-23T03:54:46.2713268Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2713383Z return func(*args, **kwargs) 2022-11-23T03:54:46.2713644Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2713750Z self.run_subtests( 2022-11-23T03:54:46.2714109Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2714260Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2714631Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2714771Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2715152Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2715267Z output = model(*input) 2022-11-23T03:54:46.2715586Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2715717Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2716101Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2716267Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2716640Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2716752Z _lazy_init(state, module) 2022-11-23T03:54:46.2717114Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2717243Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2717587Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2717757Z return func(*args, **kwargs) 2022-11-23T03:54:46.2718146Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2718239Z p_assert( 2022-11-23T03:54:46.2718579Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2718699Z traceback.print_stack() 2022-11-23T03:54:46.2718927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 0 2022-11-23T03:54:46.2719325Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:10 with 2 nodes. 2022-11-23T03:54:46.2719720Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:10 with 2 nodes. 2022-11-23T03:54:46.2719841Z File "", line 1, in 2022-11-23T03:54:46.2720102Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2720225Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2720415Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2720556Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2720757Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2720850Z self.run() 2022-11-23T03:54:46.2721043Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2721179Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2721528Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2721656Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2722024Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2722149Z getattr(self, test_name)() 2022-11-23T03:54:46.2722514Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2722607Z fn() 2022-11-23T03:54:46.2722978Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2723092Z test(self, **param_kwargs) 2022-11-23T03:54:46.2723452Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2723567Z return func(*args, **kwargs) 2022-11-23T03:54:46.2723815Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2723922Z self.run_subtests( 2022-11-23T03:54:46.2724278Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2724438Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2724808Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2724948Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2725330Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2725444Z output = model(*input) 2022-11-23T03:54:46.2725777Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2725907Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2726289Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2726453Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2726834Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2727001Z _lazy_init(state, module) 2022-11-23T03:54:46.2727360Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2727489Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2727879Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2727992Z return func(*args, **kwargs) 2022-11-23T03:54:46.2728363Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2728456Z p_assert( 2022-11-23T03:54:46.2728797Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2728915Z traceback.print_stack() 2022-11-23T03:54:46.2729190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 0 2022-11-23T03:54:46.2729315Z File "", line 1, in 2022-11-23T03:54:46.2729511Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2729642Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2729830Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2729969Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2730172Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2730271Z self.run() 2022-11-23T03:54:46.2730462Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2730593Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2730935Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2731056Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2731423Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2731526Z getattr(self, test_name)() 2022-11-23T03:54:46.2731886Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2731971Z fn() 2022-11-23T03:54:46.2732337Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2732448Z test(self, **param_kwargs) 2022-11-23T03:54:46.2732808Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2732920Z return func(*args, **kwargs) 2022-11-23T03:54:46.2733180Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2733283Z self.run_subtests( 2022-11-23T03:54:46.2733649Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2733798Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2734165Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2734309Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2734691Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2734802Z output = model(*input) 2022-11-23T03:54:46.2735135Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2735265Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2735648Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2735888Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2736253Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2736365Z _lazy_init(state, module) 2022-11-23T03:54:46.2736718Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2736852Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2737197Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2737311Z return func(*args, **kwargs) 2022-11-23T03:54:46.2737694Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2737784Z p_assert( 2022-11-23T03:54:46.2738122Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2738302Z traceback.print_stack() 2022-11-23T03:54:46.2738527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 1 2022-11-23T03:54:46.2738922Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:11 with 2 nodes. 2022-11-23T03:54:46.2739318Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:11 with 2 nodes. 2022-11-23T03:54:46.2739441Z File "", line 1, in 2022-11-23T03:54:46.2739645Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2739782Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2739972Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2740115Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2740309Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2740408Z self.run() 2022-11-23T03:54:46.2740599Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2740734Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2741079Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2741206Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2741574Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2741693Z getattr(self, test_name)() 2022-11-23T03:54:46.2742060Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2742149Z fn() 2022-11-23T03:54:46.2742522Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2742644Z test(self, **param_kwargs) 2022-11-23T03:54:46.2743008Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2743124Z return func(*args, **kwargs) 2022-11-23T03:54:46.2743388Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2743491Z self.run_subtests( 2022-11-23T03:54:46.2743849Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2743989Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2744357Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2744503Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2744891Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2745068Z output = model(*input) 2022-11-23T03:54:46.2745405Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2745539Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2745922Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2746088Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2746461Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2746574Z _lazy_init(state, module) 2022-11-23T03:54:46.2746929Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2747064Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2747453Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2747575Z return func(*args, **kwargs) 2022-11-23T03:54:46.2747960Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2748053Z p_assert( 2022-11-23T03:54:46.2748390Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2748505Z traceback.print_stack() 2022-11-23T03:54:46.2748718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 1 2022-11-23T03:54:46.2748836Z File "", line 1, in 2022-11-23T03:54:46.2749036Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2749168Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2749358Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2749511Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2749716Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2749812Z self.run() 2022-11-23T03:54:46.2750001Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2750136Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2750481Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2750604Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2750969Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2751083Z getattr(self, test_name)() 2022-11-23T03:54:46.2751450Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2751540Z fn() 2022-11-23T03:54:46.2751903Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2752018Z test(self, **param_kwargs) 2022-11-23T03:54:46.2752379Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2752497Z return func(*args, **kwargs) 2022-11-23T03:54:46.2752760Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2752867Z self.run_subtests( 2022-11-23T03:54:46.2753228Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2753382Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2753751Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2753954Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2754338Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2754445Z output = model(*input) 2022-11-23T03:54:46.2754775Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2754909Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2755294Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2755461Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2755836Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2755953Z _lazy_init(state, module) 2022-11-23T03:54:46.2756355Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2756480Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2756827Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2756942Z return func(*args, **kwargs) 2022-11-23T03:54:46.2757326Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2757419Z p_assert( 2022-11-23T03:54:46.2757757Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2757875Z traceback.print_stack() 2022-11-23T03:54:46.2758099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 0 2022-11-23T03:54:46.2758492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:12 with 2 nodes. 2022-11-23T03:54:46.2758894Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:12 with 2 nodes. 2022-11-23T03:54:46.2759027Z File "", line 1, in 2022-11-23T03:54:46.2759224Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2759352Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2759542Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2759684Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2759888Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2759982Z self.run() 2022-11-23T03:54:46.2760174Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2760295Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2760642Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2760771Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2761139Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2761256Z getattr(self, test_name)() 2022-11-23T03:54:46.2761623Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2761711Z fn() 2022-11-23T03:54:46.2762082Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2762200Z test(self, **param_kwargs) 2022-11-23T03:54:46.2762562Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2762678Z return func(*args, **kwargs) 2022-11-23T03:54:46.2762939Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2763107Z self.run_subtests( 2022-11-23T03:54:46.2763470Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2763620Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2763990Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2764134Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2764515Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2764611Z output = model(*input) 2022-11-23T03:54:46.2764945Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2765077Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2765507Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2765678Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2766055Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2766169Z _lazy_init(state, module) 2022-11-23T03:54:46.2766526Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2766660Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2767000Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2767115Z return func(*args, **kwargs) 2022-11-23T03:54:46.2767501Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2767596Z p_assert( 2022-11-23T03:54:46.2768050Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2768169Z traceback.print_stack() 2022-11-23T03:54:46.2768396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 0 2022-11-23T03:54:46.2768515Z File "", line 1, in 2022-11-23T03:54:46.2768716Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2768835Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2769027Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2769168Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2769369Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2769462Z self.run() 2022-11-23T03:54:46.2769657Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2769793Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2770145Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2770267Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2770636Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2770751Z getattr(self, test_name)() 2022-11-23T03:54:46.2771113Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2771199Z fn() 2022-11-23T03:54:46.2771569Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2771684Z test(self, **param_kwargs) 2022-11-23T03:54:46.2772046Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2772160Z return func(*args, **kwargs) 2022-11-23T03:54:46.2772480Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2772582Z self.run_subtests( 2022-11-23T03:54:46.2772941Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2773095Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2773462Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2773603Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2773984Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2774094Z output = model(*input) 2022-11-23T03:54:46.2774424Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2774605Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2774995Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2775157Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2775529Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2775642Z _lazy_init(state, module) 2022-11-23T03:54:46.2775999Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2776129Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2776470Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2776589Z return func(*args, **kwargs) 2022-11-23T03:54:46.2776976Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2777059Z p_assert( 2022-11-23T03:54:46.2777398Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2777511Z traceback.print_stack() 2022-11-23T03:54:46.2777742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 1 2022-11-23T03:54:46.2778140Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:13 with 2 nodes. 2022-11-23T03:54:46.2778534Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:13 with 2 nodes. 2022-11-23T03:54:46.2778654Z File "", line 1, in 2022-11-23T03:54:46.2778852Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2778986Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2779178Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2779326Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2779530Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2779625Z self.run() 2022-11-23T03:54:46.2779815Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2779948Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2780292Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2780417Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2780772Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2780887Z getattr(self, test_name)() 2022-11-23T03:54:46.2781251Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2781397Z fn() 2022-11-23T03:54:46.2781774Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2781888Z test(self, **param_kwargs) 2022-11-23T03:54:46.2782255Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2782372Z return func(*args, **kwargs) 2022-11-23T03:54:46.2782636Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2782740Z self.run_subtests( 2022-11-23T03:54:46.2783098Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2783252Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2783620Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2783814Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2784203Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2784315Z output = model(*input) 2022-11-23T03:54:46.2784648Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2784781Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2785165Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2785318Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2785692Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2785807Z _lazy_init(state, module) 2022-11-23T03:54:46.2786168Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2786302Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2786646Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2786761Z return func(*args, **kwargs) 2022-11-23T03:54:46.2787143Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2787236Z p_assert( 2022-11-23T03:54:46.2787575Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2787690Z traceback.print_stack() 2022-11-23T03:54:46.2787913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 1 2022-11-23T03:54:46.2788033Z File "", line 1, in 2022-11-23T03:54:46.2788232Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2788374Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2788563Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2788704Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2788892Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2788992Z self.run() 2022-11-23T03:54:46.2789184Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2789321Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2789667Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2789791Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2790158Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2790277Z getattr(self, test_name)() 2022-11-23T03:54:46.2790716Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2790805Z fn() 2022-11-23T03:54:46.2791176Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2791294Z test(self, **param_kwargs) 2022-11-23T03:54:46.2791653Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2791770Z return func(*args, **kwargs) 2022-11-23T03:54:46.2792031Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 258, in test_mixture_of_experts_with_delay_before_free 2022-11-23T03:54:46.2792139Z self.run_subtests( 2022-11-23T03:54:46.2792496Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2792647Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2793049Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2793195Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2793583Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2793691Z output = model(*input) 2022-11-23T03:54:46.2794024Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2794155Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2794535Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2794701Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2795074Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2795192Z _lazy_init(state, module) 2022-11-23T03:54:46.2795549Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.2795680Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2796021Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2796139Z return func(*args, **kwargs) 2022-11-23T03:54:46.2796521Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2796613Z p_assert( 2022-11-23T03:54:46.2796952Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2797065Z traceback.print_stack() 2022-11-23T03:54:46.2797276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 0 2022-11-23T03:54:46.2797675Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:14 with 2 nodes. 2022-11-23T03:54:46.2798068Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:14 with 2 nodes. 2022-11-23T03:54:46.2798290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 1 2022-11-23T03:54:46.2798509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 0 2022-11-23T03:54:46.2798905Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:15 with 2 nodes. 2022-11-23T03:54:46.2799299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:15 with 2 nodes. 2022-11-23T03:54:46.2799519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 0 2022-11-23T03:54:46.2799911Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:16 with 2 nodes. 2022-11-23T03:54:46.2800196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 1 2022-11-23T03:54:46.2800588Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:16 with 2 nodes. 2022-11-23T03:54:46.2801353Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2801578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 1 2022-11-23T03:54:46.2801798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 0 2022-11-23T03:54:46.2802234Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:17 with 2 nodes. 2022-11-23T03:54:46.2802630Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:17 with 2 nodes. 2022-11-23T03:54:46.2802858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 1 2022-11-23T03:54:46.2803080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 0 2022-11-23T03:54:46.2803472Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:18 with 2 nodes. 2022-11-23T03:54:46.2803859Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:18 with 2 nodes. 2022-11-23T03:54:46.2804634Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2804856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 0 2022-11-23T03:54:46.2805075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 1 2022-11-23T03:54:46.2805464Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:19 with 2 nodes. 2022-11-23T03:54:46.2805855Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:19 with 2 nodes. 2022-11-23T03:54:46.2806617Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2807381Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2807604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:20 to store for rank: 1 2022-11-23T03:54:46.2807874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:20 to store for rank: 0 2022-11-23T03:54:46.2808268Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:20 with 2 nodes. 2022-11-23T03:54:46.2808659Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:20 with 2 nodes. 2022-11-23T03:54:46.2809423Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2809719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:21 to store for rank: 1 2022-11-23T03:54:46.2809944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:21 to store for rank: 0 2022-11-23T03:54:46.2810339Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:21 with 2 nodes. 2022-11-23T03:54:46.2810729Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:21 with 2 nodes. 2022-11-23T03:54:46.2810955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:22 to store for rank: 1 2022-11-23T03:54:46.2811227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:22 to store for rank: 0 2022-11-23T03:54:46.2811628Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:22 with 2 nodes. 2022-11-23T03:54:46.2812015Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:22 with 2 nodes. 2022-11-23T03:54:46.2812773Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2812999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:23 to store for rank: 0 2022-11-23T03:54:46.2813218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:23 to store for rank: 1 2022-11-23T03:54:46.2813598Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:23 with 2 nodes. 2022-11-23T03:54:46.2813993Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:23 with 2 nodes. 2022-11-23T03:54:46.2814217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:24 to store for rank: 1 2022-11-23T03:54:46.2814436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:24 to store for rank: 0 2022-11-23T03:54:46.2814824Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:24 with 2 nodes. 2022-11-23T03:54:46.2815217Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:24 with 2 nodes. 2022-11-23T03:54:46.2815989Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2816219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:25 to store for rank: 1 2022-11-23T03:54:46.2816443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:25 to store for rank: 0 2022-11-23T03:54:46.2816835Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:25 with 2 nodes. 2022-11-23T03:54:46.2817223Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:25 with 2 nodes. 2022-11-23T03:54:46.2817448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:26 to store for rank: 0 2022-11-23T03:54:46.2817839Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:26 with 2 nodes. 2022-11-23T03:54:46.2818116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:26 to store for rank: 1 2022-11-23T03:54:46.2818509Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:26 with 2 nodes. 2022-11-23T03:54:46.2819281Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2819504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:27 to store for rank: 1 2022-11-23T03:54:46.2819723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:27 to store for rank: 0 2022-11-23T03:54:46.2820114Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:27 with 2 nodes. 2022-11-23T03:54:46.2820552Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:27 with 2 nodes. 2022-11-23T03:54:46.2820775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:28 to store for rank: 1 2022-11-23T03:54:46.2820996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:28 to store for rank: 0 2022-11-23T03:54:46.2821386Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:28 with 2 nodes. 2022-11-23T03:54:46.2821774Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:28 with 2 nodes. 2022-11-23T03:54:46.2821992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:29 to store for rank: 1 2022-11-23T03:54:46.2822746Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2822972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:29 to store for rank: 0 2022-11-23T03:54:46.2823359Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:29 with 2 nodes. 2022-11-23T03:54:46.2823749Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:29 with 2 nodes. 2022-11-23T03:54:46.2823974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:30 to store for rank: 1 2022-11-23T03:54:46.2824365Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:30 with 2 nodes. 2022-11-23T03:54:46.2824588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:30 to store for rank: 0 2022-11-23T03:54:46.2824981Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:30 with 2 nodes. 2022-11-23T03:54:46.2825740Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2825965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:31 to store for rank: 1 2022-11-23T03:54:46.2826188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:31 to store for rank: 0 2022-11-23T03:54:46.2826581Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:31 with 2 nodes. 2022-11-23T03:54:46.2826971Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:31 with 2 nodes. 2022-11-23T03:54:46.2827273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:32 to store for rank: 1 2022-11-23T03:54:46.2827483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:32 to store for rank: 0 2022-11-23T03:54:46.2827880Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:32 with 2 nodes. 2022-11-23T03:54:46.2828270Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:32 with 2 nodes. 2022-11-23T03:54:46.2829029Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2829301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:33 to store for rank: 1 2022-11-23T03:54:46.2829528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:33 to store for rank: 0 2022-11-23T03:54:46.2829916Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:33 with 2 nodes. 2022-11-23T03:54:46.2830304Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:33 with 2 nodes. 2022-11-23T03:54:46.2830525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:34 to store for rank: 0 2022-11-23T03:54:46.2830743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:34 to store for rank: 1 2022-11-23T03:54:46.2831131Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:34 with 2 nodes. 2022-11-23T03:54:46.2831521Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:34 with 2 nodes. 2022-11-23T03:54:46.2832283Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2833038Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2833261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:35 to store for rank: 0 2022-11-23T03:54:46.2833480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:35 to store for rank: 1 2022-11-23T03:54:46.2833873Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:35 with 2 nodes. 2022-11-23T03:54:46.2834265Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:35 with 2 nodes. 2022-11-23T03:54:46.2835025Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2835249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:36 to store for rank: 1 2022-11-23T03:54:46.2835471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:36 to store for rank: 0 2022-11-23T03:54:46.2835860Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:36 with 2 nodes. 2022-11-23T03:54:46.2836301Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:36 with 2 nodes. 2022-11-23T03:54:46.2837061Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2837285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:37 to store for rank: 1 2022-11-23T03:54:46.2837507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:37 to store for rank: 0 2022-11-23T03:54:46.2837897Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:37 with 2 nodes. 2022-11-23T03:54:46.2838288Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:37 with 2 nodes. 2022-11-23T03:54:46.2838437Z dist init r=1, world=2 2022-11-23T03:54:46.2838754Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.2839063Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.2839367Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.2839670Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.2839973Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.2840279Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.2840581Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.2840880Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.2841178Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.2841476Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.2841777Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.2842078Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.2842180Z dist init r=0, world=2 2022-11-23T03:54:46.2842489Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.2842793Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.2843099Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.2843404Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.2843750Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.2844047Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.2844348Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.2844645Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.2844979Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.2845286Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.2845586Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.2845884Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.2845964Z ok (54.616s) 2022-11-23T03:54:46.2846301Z test_nested_always_wrap_model_offload_false_no_shard_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48258 2022-11-23T03:54:46.2846511Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48259 2022-11-23T03:54:46.2846907Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.2847075Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.2847462Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.2847640Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.2847907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.2848285Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.2848449Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.2848832Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.2849014Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.2849238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.2849628Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.2850018Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.2850294Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.2850573Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.2850789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.2851004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.2851222Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2851509Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2852570Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.2852673Z warnings.warn( 2022-11-23T03:54:46.2852895Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2853986Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.2854092Z warnings.warn( 2022-11-23T03:54:46.2854307Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2854525Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2854729Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2854948Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2855162Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2855383Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2855602Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2855814Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2856029Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2856243Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2856462Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2856678Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2856893Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2857106Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2857319Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2857537Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2857756Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2857971Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2858185Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2858954Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2859709Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2860521Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2861282Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2862081Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2862843Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2863602Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2864364Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2865122Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2865879Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2866096Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2866318Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2866539Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2866757Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2866976Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2867190Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2867408Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2867622Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2867838Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2868053Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2868273Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2868534Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2868750Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2868964Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2869179Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2869391Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2869593Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2869810Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2870028Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2870300Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2870523Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2870734Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2870951Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2871166Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2871935Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2872697Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2873464Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2874217Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2874975Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2875733Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2876500Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2877254Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2878064Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2878820Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.2879039Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2879296Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2879404Z dist init r=0, world=2 2022-11-23T03:54:46.2879507Z dist init r=1, world=2 2022-11-23T03:54:46.2879587Z ok (8.638s) 2022-11-23T03:54:46.2879919Z test_nested_always_wrap_model_offload_false_none_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48411 2022-11-23T03:54:46.2880130Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48412 2022-11-23T03:54:46.2880511Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.2880677Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.2881061Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.2881241Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.2881472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.2881844Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.2882007Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.2882395Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.2882576Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.2882799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.2883195Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.2883590Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.2883869Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.2884150Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.2884364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.2884581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.2884797Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2885016Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2886070Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.2886230Z warnings.warn( 2022-11-23T03:54:46.2886448Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2887501Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.2887603Z warnings.warn( 2022-11-23T03:54:46.2887935Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2888204Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2888424Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2888642Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2888859Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2889073Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2889291Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2889506Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2889719Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2889934Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2890151Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2890368Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2890583Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2890796Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2891010Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2891224Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2891437Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2891653Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2891867Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2892084Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2892294Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2892494Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2892706Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2892923Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2893139Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2893356Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2893567Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2893783Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2894049Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2894269Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2894481Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2894695Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2894907Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2895121Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2895335Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2895548Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2895802Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2896018Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2896231Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2896446Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2896657Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2896873Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2897072Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2897286Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2897498Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2897608Z dist init r=0, world=2 2022-11-23T03:54:46.2897711Z dist init r=1, world=2 2022-11-23T03:54:46.2897805Z ok (8.640s) 2022-11-23T03:54:46.2898154Z test_nested_always_wrap_model_offload_false_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48564 2022-11-23T03:54:46.2898363Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48565 2022-11-23T03:54:46.2898753Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.2898917Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.2899301Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.2899484Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.2899714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.2900089Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.2900253Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.2900641Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.2900819Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.2901042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.2901435Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.2901827Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.2902096Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.2902432Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.2902648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.2902864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.2903082Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2903295Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2904379Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.2904485Z warnings.warn( 2022-11-23T03:54:46.2904703Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2905743Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.2905849Z warnings.warn( 2022-11-23T03:54:46.2906063Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2906281Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2906501Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2906714Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2906927Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2907145Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2907360Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2907570Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2907786Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2907999Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2908217Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2908434Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2908646Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2908860Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2909072Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2909290Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2909489Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2909703Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2909916Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2910133Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2910398Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2910614Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2910825Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2911039Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2911252Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2911467Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2911684Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2911898Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2912149Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2912368Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2912580Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2912795Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2913008Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2913222Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2913439Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2913652Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2913852Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2914074Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2914286Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2914503Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2914717Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2914929Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2915142Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2915355Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2915567Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2915671Z dist init r=0, world=2 2022-11-23T03:54:46.2915780Z dist init r=1, world=2 2022-11-23T03:54:46.2915876Z ok (8.437s) 2022-11-23T03:54:46.2916215Z test_nested_always_wrap_model_offload_true_no_shard_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48717 2022-11-23T03:54:46.2916428Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48718 2022-11-23T03:54:46.2916815Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.2916984Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.2917370Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.2917548Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.2917773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.2918210Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.2918363Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.2918746Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.2918923Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.2919150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.2919548Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.2919939Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.2920215Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.2920535Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.2920757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.2920972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.2921188Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2921403Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2922463Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.2922572Z warnings.warn( 2022-11-23T03:54:46.2922693Z File "", line 1, in 2022-11-23T03:54:46.2922894Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2923029Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2923220Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2923366Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2923571Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2923670Z self.run() 2022-11-23T03:54:46.2923865Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2923989Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2924337Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2924461Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2924835Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2924952Z getattr(self, test_name)() 2022-11-23T03:54:46.2925316Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2925407Z fn() 2022-11-23T03:54:46.2925781Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2925898Z test(self, **param_kwargs) 2022-11-23T03:54:46.2926258Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2926373Z return func(*args, **kwargs) 2022-11-23T03:54:46.2926615Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.2926720Z self.run_subtests( 2022-11-23T03:54:46.2927144Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2927300Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2927671Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2927859Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2928243Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2928340Z output = model(*input) 2022-11-23T03:54:46.2928676Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2928809Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2929190Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2929420Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2929803Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2929918Z _lazy_init(state, module) 2022-11-23T03:54:46.2930274Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.2930406Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2930750Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2930866Z return func(*args, **kwargs) 2022-11-23T03:54:46.2931249Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2931348Z p_assert( 2022-11-23T03:54:46.2931686Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2931812Z traceback.print_stack() 2022-11-23T03:54:46.2932033Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2933081Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.2933183Z warnings.warn( 2022-11-23T03:54:46.2933304Z File "", line 1, in 2022-11-23T03:54:46.2933503Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2933622Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2933818Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2933962Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2934167Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2934266Z self.run() 2022-11-23T03:54:46.2934461Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2934595Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2934936Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2935062Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2935434Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2935547Z getattr(self, test_name)() 2022-11-23T03:54:46.2935913Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2936092Z fn() 2022-11-23T03:54:46.2936466Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2936581Z test(self, **param_kwargs) 2022-11-23T03:54:46.2936947Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2937062Z return func(*args, **kwargs) 2022-11-23T03:54:46.2937290Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.2937394Z self.run_subtests( 2022-11-23T03:54:46.2937755Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2937907Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2938282Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2938473Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2938863Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2938972Z output = model(*input) 2022-11-23T03:54:46.2939304Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2939437Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2939823Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2939989Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2940364Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2940477Z _lazy_init(state, module) 2022-11-23T03:54:46.2940838Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.2940977Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2941322Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2941439Z return func(*args, **kwargs) 2022-11-23T03:54:46.2941825Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2941906Z p_assert( 2022-11-23T03:54:46.2942245Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2942359Z traceback.print_stack() 2022-11-23T03:54:46.2942580Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2942705Z File "", line 1, in 2022-11-23T03:54:46.2942905Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2943044Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2943233Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2943372Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2943576Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2943671Z self.run() 2022-11-23T03:54:46.2943862Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2944000Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2944346Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2944470Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2944843Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2944960Z getattr(self, test_name)() 2022-11-23T03:54:46.2945385Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2945475Z fn() 2022-11-23T03:54:46.2945848Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2945963Z test(self, **param_kwargs) 2022-11-23T03:54:46.2946324Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2946446Z return func(*args, **kwargs) 2022-11-23T03:54:46.2946694Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.2946798Z self.run_subtests( 2022-11-23T03:54:46.2947155Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2947307Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2947721Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2947867Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2948251Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2948366Z output = model(*input) 2022-11-23T03:54:46.2948699Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2948836Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2949223Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2949391Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2949750Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2949866Z _lazy_init(state, module) 2022-11-23T03:54:46.2950222Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.2950355Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2950704Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2950817Z return func(*args, **kwargs) 2022-11-23T03:54:46.2951203Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2951298Z p_assert( 2022-11-23T03:54:46.2951640Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2951754Z traceback.print_stack() 2022-11-23T03:54:46.2951971Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2952090Z File "", line 1, in 2022-11-23T03:54:46.2952293Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2952428Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2952618Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2952757Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2952960Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2953042Z self.run() 2022-11-23T03:54:46.2953234Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2953370Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2953715Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2953841Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2954214Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2954392Z getattr(self, test_name)() 2022-11-23T03:54:46.2954765Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2954858Z fn() 2022-11-23T03:54:46.2955228Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2955342Z test(self, **param_kwargs) 2022-11-23T03:54:46.2955706Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2955824Z return func(*args, **kwargs) 2022-11-23T03:54:46.2956067Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.2956174Z self.run_subtests( 2022-11-23T03:54:46.2956532Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2956733Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2957107Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2957237Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2957619Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2957728Z output = model(*input) 2022-11-23T03:54:46.2958059Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2958189Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2958574Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2958740Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2959123Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2959240Z _lazy_init(state, module) 2022-11-23T03:54:46.2959599Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.2959732Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2960079Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2960194Z return func(*args, **kwargs) 2022-11-23T03:54:46.2960580Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2960672Z p_assert( 2022-11-23T03:54:46.2961015Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2961130Z traceback.print_stack() 2022-11-23T03:54:46.2961351Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2961459Z File "", line 1, in 2022-11-23T03:54:46.2961660Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2961793Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2961984Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2962126Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2962329Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2962423Z self.run() 2022-11-23T03:54:46.2962619Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2962759Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2963107Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2963233Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2963671Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2963786Z getattr(self, test_name)() 2022-11-23T03:54:46.2964155Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2964246Z fn() 2022-11-23T03:54:46.2964616Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2964732Z test(self, **param_kwargs) 2022-11-23T03:54:46.2965083Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2965199Z return func(*args, **kwargs) 2022-11-23T03:54:46.2965442Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.2965547Z self.run_subtests( 2022-11-23T03:54:46.2965955Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2966111Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2966486Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2966631Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2967012Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2967123Z output = model(*input) 2022-11-23T03:54:46.2967457Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2967593Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2968094Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2968268Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2968643Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2968757Z _lazy_init(state, module) 2022-11-23T03:54:46.2969112Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.2969243Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2969574Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2969688Z return func(*args, **kwargs) 2022-11-23T03:54:46.2970072Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2970167Z p_assert( 2022-11-23T03:54:46.2970508Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2970631Z traceback.print_stack() 2022-11-23T03:54:46.2970848Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2970971Z File "", line 1, in 2022-11-23T03:54:46.2971167Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2971301Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2971491Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2971632Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2971832Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2971928Z self.run() 2022-11-23T03:54:46.2972117Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2972249Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2972596Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2972778Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2973151Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2973265Z getattr(self, test_name)() 2022-11-23T03:54:46.2973628Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2973717Z fn() 2022-11-23T03:54:46.2974088Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2974200Z test(self, **param_kwargs) 2022-11-23T03:54:46.2974562Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2974675Z return func(*args, **kwargs) 2022-11-23T03:54:46.2974968Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.2975077Z self.run_subtests( 2022-11-23T03:54:46.2975435Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2975589Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2975957Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2976104Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2976490Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2976600Z output = model(*input) 2022-11-23T03:54:46.2976932Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2977051Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2977438Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2977607Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2977980Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2978097Z _lazy_init(state, module) 2022-11-23T03:54:46.2978454Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.2978585Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2978927Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2979043Z return func(*args, **kwargs) 2022-11-23T03:54:46.2979427Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2979517Z p_assert( 2022-11-23T03:54:46.2979861Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2979976Z traceback.print_stack() 2022-11-23T03:54:46.2980191Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2980315Z File "", line 1, in 2022-11-23T03:54:46.2980512Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2980646Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2980838Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2980965Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2981166Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2981264Z self.run() 2022-11-23T03:54:46.2981455Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2981656Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2982007Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2982133Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2982501Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2982617Z getattr(self, test_name)() 2022-11-23T03:54:46.2982981Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2983069Z fn() 2022-11-23T03:54:46.2983440Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2983557Z test(self, **param_kwargs) 2022-11-23T03:54:46.2983921Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2984036Z return func(*args, **kwargs) 2022-11-23T03:54:46.2984326Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.2984433Z self.run_subtests( 2022-11-23T03:54:46.2984781Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2984933Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2985299Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2985441Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2985823Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2985935Z output = model(*input) 2022-11-23T03:54:46.2986268Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2986406Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2986790Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2986955Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2987330Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2987445Z _lazy_init(state, module) 2022-11-23T03:54:46.2987803Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.2987933Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2988274Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2988392Z return func(*args, **kwargs) 2022-11-23T03:54:46.2988777Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2988874Z p_assert( 2022-11-23T03:54:46.2989200Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2989314Z traceback.print_stack() 2022-11-23T03:54:46.2989531Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2989649Z File "", line 1, in 2022-11-23T03:54:46.2989845Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2989977Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2990167Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2990306Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2990508Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.2990602Z self.run() 2022-11-23T03:54:46.2990864Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.2991000Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.2991348Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.2991473Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.2991839Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.2991955Z getattr(self, test_name)() 2022-11-23T03:54:46.2992318Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.2992395Z fn() 2022-11-23T03:54:46.2992768Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.2992888Z test(self, **param_kwargs) 2022-11-23T03:54:46.2993315Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.2993436Z return func(*args, **kwargs) 2022-11-23T03:54:46.2993679Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.2993797Z self.run_subtests( 2022-11-23T03:54:46.2994177Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.2994333Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.2994702Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.2994847Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.2995230Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.2995340Z output = model(*input) 2022-11-23T03:54:46.2995676Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.2995810Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.2996194Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.2996360Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.2996733Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.2996832Z _lazy_init(state, module) 2022-11-23T03:54:46.2997188Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.2997320Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.2997666Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.2997782Z return func(*args, **kwargs) 2022-11-23T03:54:46.2998171Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.2998265Z p_assert( 2022-11-23T03:54:46.2998604Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.2998718Z traceback.print_stack() 2022-11-23T03:54:46.2998937Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.2999056Z File "", line 1, in 2022-11-23T03:54:46.2999250Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.2999385Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.2999577Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.2999718Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.2999924Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3000089Z self.run() 2022-11-23T03:54:46.3000267Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3000400Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3000746Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3000870Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3001239Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3001352Z getattr(self, test_name)() 2022-11-23T03:54:46.3001715Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3001804Z fn() 2022-11-23T03:54:46.3002175Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3002354Z test(self, **param_kwargs) 2022-11-23T03:54:46.3002724Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3002842Z return func(*args, **kwargs) 2022-11-23T03:54:46.3003083Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3003186Z self.run_subtests( 2022-11-23T03:54:46.3003543Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3003696Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3004063Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3004208Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3004580Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3004698Z output = model(*input) 2022-11-23T03:54:46.3005028Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3005163Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3005549Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3005715Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3006085Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3006198Z _lazy_init(state, module) 2022-11-23T03:54:46.3006556Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3006686Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3007035Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3007151Z return func(*args, **kwargs) 2022-11-23T03:54:46.3007535Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3007630Z p_assert( 2022-11-23T03:54:46.3008016Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3008129Z traceback.print_stack() 2022-11-23T03:54:46.3008346Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3008466Z File "", line 1, in 2022-11-23T03:54:46.3008649Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3008782Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3008973Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3009197Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3009399Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3009495Z self.run() 2022-11-23T03:54:46.3009686Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3009819Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3010168Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3010292Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3010664Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3010779Z getattr(self, test_name)() 2022-11-23T03:54:46.3011142Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3011233Z fn() 2022-11-23T03:54:46.3011654Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3011774Z test(self, **param_kwargs) 2022-11-23T03:54:46.3012141Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3012244Z return func(*args, **kwargs) 2022-11-23T03:54:46.3012483Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3012588Z self.run_subtests( 2022-11-23T03:54:46.3012946Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3013103Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3013471Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3013615Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3014004Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3014115Z output = model(*input) 2022-11-23T03:54:46.3014446Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3014580Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3014961Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3015125Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3015498Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3015609Z _lazy_init(state, module) 2022-11-23T03:54:46.3015966Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3016106Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3016447Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3016562Z return func(*args, **kwargs) 2022-11-23T03:54:46.3016934Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3017028Z p_assert( 2022-11-23T03:54:46.3017364Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3017479Z traceback.print_stack() 2022-11-23T03:54:46.3017699Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3017816Z File "", line 1, in 2022-11-23T03:54:46.3018012Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3018151Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3018404Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3018546Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3018747Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3018841Z self.run() 2022-11-23T03:54:46.3019035Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3019170Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3019517Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3019639Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3019995Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3020111Z getattr(self, test_name)() 2022-11-23T03:54:46.3020521Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3020620Z fn() 2022-11-23T03:54:46.3020994Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3021110Z test(self, **param_kwargs) 2022-11-23T03:54:46.3021475Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3021592Z return func(*args, **kwargs) 2022-11-23T03:54:46.3021834Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3021939Z self.run_subtests( 2022-11-23T03:54:46.3022298Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3022447Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3022816Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3022964Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3023346Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3023459Z output = model(*input) 2022-11-23T03:54:46.3023793Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3023926Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3024294Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3024458Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3024829Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3024945Z _lazy_init(state, module) 2022-11-23T03:54:46.3025309Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3025441Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3025787Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3025905Z return func(*args, **kwargs) 2022-11-23T03:54:46.3026289Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3026383Z p_assert( 2022-11-23T03:54:46.3026722Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3026840Z traceback.print_stack() 2022-11-23T03:54:46.3027055Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3027177Z File "", line 1, in 2022-11-23T03:54:46.3027377Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3027573Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3027764Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3027904Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3028093Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3028192Z self.run() 2022-11-23T03:54:46.3028385Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3028517Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3028866Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3028990Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3029359Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3029472Z getattr(self, test_name)() 2022-11-23T03:54:46.3029886Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3029979Z fn() 2022-11-23T03:54:46.3030352Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3030464Z test(self, **param_kwargs) 2022-11-23T03:54:46.3030830Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3030946Z return func(*args, **kwargs) 2022-11-23T03:54:46.3031187Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3031290Z self.run_subtests( 2022-11-23T03:54:46.3031647Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3031787Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3032165Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3032309Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3032690Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3032803Z output = model(*input) 2022-11-23T03:54:46.3033138Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3033271Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3033655Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3033820Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3034199Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3034322Z _lazy_init(state, module) 2022-11-23T03:54:46.3034679Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3034810Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3035154Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3035270Z return func(*args, **kwargs) 2022-11-23T03:54:46.3035655Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3035748Z p_assert( 2022-11-23T03:54:46.3036090Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3036193Z traceback.print_stack() 2022-11-23T03:54:46.3036412Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3036595Z File "", line 1, in 2022-11-23T03:54:46.3036795Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3036928Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3037119Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3037260Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3037466Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3037562Z self.run() 2022-11-23T03:54:46.3037756Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3037890Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3038238Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3038368Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3038783Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3038905Z getattr(self, test_name)() 2022-11-23T03:54:46.3039274Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3039368Z fn() 2022-11-23T03:54:46.3039725Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3039840Z test(self, **param_kwargs) 2022-11-23T03:54:46.3040203Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3040317Z return func(*args, **kwargs) 2022-11-23T03:54:46.3040561Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3040667Z self.run_subtests( 2022-11-23T03:54:46.3041031Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3041188Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3041559Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3041704Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3042084Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3042198Z output = model(*input) 2022-11-23T03:54:46.3042530Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3042663Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3043044Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3043214Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3043594Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3043709Z _lazy_init(state, module) 2022-11-23T03:54:46.3044068Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3044188Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3044530Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3044643Z return func(*args, **kwargs) 2022-11-23T03:54:46.3045026Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3045120Z p_assert( 2022-11-23T03:54:46.3045458Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3045576Z traceback.print_stack() 2022-11-23T03:54:46.3045855Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3045976Z File "", line 1, in 2022-11-23T03:54:46.3046173Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3046305Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3046501Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3046646Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3046846Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3046943Z self.run() 2022-11-23T03:54:46.3047136Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3047257Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3047606Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3047829Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3048208Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3048325Z getattr(self, test_name)() 2022-11-23T03:54:46.3048692Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3048784Z fn() 2022-11-23T03:54:46.3049155Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3049272Z test(self, **param_kwargs) 2022-11-23T03:54:46.3049633Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3049750Z return func(*args, **kwargs) 2022-11-23T03:54:46.3049992Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3050100Z self.run_subtests( 2022-11-23T03:54:46.3050458Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3050611Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3050979Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3051123Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3051502Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3051599Z output = model(*input) 2022-11-23T03:54:46.3051930Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3052060Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3052443Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3052615Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3052990Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3053103Z _lazy_init(state, module) 2022-11-23T03:54:46.3053462Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3053593Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3053936Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3054051Z return func(*args, **kwargs) 2022-11-23T03:54:46.3054434Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3054524Z p_assert( 2022-11-23T03:54:46.3054867Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3055049Z traceback.print_stack() 2022-11-23T03:54:46.3055266Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3055385Z File "", line 1, in 2022-11-23T03:54:46.3055584Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3055703Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3055895Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3056040Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3056246Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3056340Z self.run() 2022-11-23T03:54:46.3056534Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3056674Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3057070Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3057200Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3057569Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3057686Z getattr(self, test_name)() 2022-11-23T03:54:46.3058050Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3058143Z fn() 2022-11-23T03:54:46.3058516Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3058632Z test(self, **param_kwargs) 2022-11-23T03:54:46.3058993Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3059110Z return func(*args, **kwargs) 2022-11-23T03:54:46.3059340Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3059448Z self.run_subtests( 2022-11-23T03:54:46.3059800Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3059950Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3060318Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3060464Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3060844Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3060953Z output = model(*input) 2022-11-23T03:54:46.3061289Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3061416Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3061804Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3061972Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3062343Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3062455Z _lazy_init(state, module) 2022-11-23T03:54:46.3062812Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3062944Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3063285Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3063404Z return func(*args, **kwargs) 2022-11-23T03:54:46.3063788Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3063931Z p_assert( 2022-11-23T03:54:46.3064276Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3064393Z traceback.print_stack() 2022-11-23T03:54:46.3064614Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3064733Z File "", line 1, in 2022-11-23T03:54:46.3064933Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3065067Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3065257Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3065401Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3065603Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3065698Z self.run() 2022-11-23T03:54:46.3065889Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3066073Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3066423Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3066550Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3066917Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3067019Z getattr(self, test_name)() 2022-11-23T03:54:46.3067383Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3067471Z fn() 2022-11-23T03:54:46.3067842Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3067957Z test(self, **param_kwargs) 2022-11-23T03:54:46.3068316Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3068441Z return func(*args, **kwargs) 2022-11-23T03:54:46.3068695Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3068799Z self.run_subtests( 2022-11-23T03:54:46.3069141Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3069296Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3069666Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3069809Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3070193Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3070304Z output = model(*input) 2022-11-23T03:54:46.3070642Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3070777Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3071160Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3071324Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3071699Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3071819Z _lazy_init(state, module) 2022-11-23T03:54:46.3072175Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3072310Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3072658Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3072776Z return func(*args, **kwargs) 2022-11-23T03:54:46.3073165Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3073350Z p_assert( 2022-11-23T03:54:46.3073693Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3073796Z traceback.print_stack() 2022-11-23T03:54:46.3074016Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3074138Z File "", line 1, in 2022-11-23T03:54:46.3074338Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3074472Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3074661Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3074806Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3075008Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3075106Z self.run() 2022-11-23T03:54:46.3075343Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3075483Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3075835Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3075963Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3076335Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3076451Z getattr(self, test_name)() 2022-11-23T03:54:46.3076821Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3076897Z fn() 2022-11-23T03:54:46.3077268Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3077384Z test(self, **param_kwargs) 2022-11-23T03:54:46.3077755Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3077871Z return func(*args, **kwargs) 2022-11-23T03:54:46.3078112Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3078217Z self.run_subtests( 2022-11-23T03:54:46.3078578Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3078732Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3079103Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3079245Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3079626Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3079736Z output = model(*input) 2022-11-23T03:54:46.3080078Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3080210Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3080595Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3080760Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3081135Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3081247Z _lazy_init(state, module) 2022-11-23T03:54:46.3081591Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3081726Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3082070Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3082253Z return func(*args, **kwargs) 2022-11-23T03:54:46.3082643Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3082737Z p_assert( 2022-11-23T03:54:46.3083077Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3083195Z traceback.print_stack() 2022-11-23T03:54:46.3083415Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3083536Z File "", line 1, in 2022-11-23T03:54:46.3083735Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3083870Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3084065Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3084208Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3084462Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3084561Z self.run() 2022-11-23T03:54:46.3084756Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3084878Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3085228Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3085354Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3085725Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3085841Z getattr(self, test_name)() 2022-11-23T03:54:46.3086206Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3086296Z fn() 2022-11-23T03:54:46.3086674Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3086797Z test(self, **param_kwargs) 2022-11-23T03:54:46.3087162Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3087281Z return func(*args, **kwargs) 2022-11-23T03:54:46.3087523Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3087627Z self.run_subtests( 2022-11-23T03:54:46.3088095Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3088248Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3088619Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3088763Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3089150Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3089251Z output = model(*input) 2022-11-23T03:54:46.3089585Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3089720Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3090103Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3090268Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3090641Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3090754Z _lazy_init(state, module) 2022-11-23T03:54:46.3091112Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3091246Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3091673Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3091790Z return func(*args, **kwargs) 2022-11-23T03:54:46.3092175Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3092272Z p_assert( 2022-11-23T03:54:46.3092613Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3092735Z traceback.print_stack() 2022-11-23T03:54:46.3092958Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3093078Z File "", line 1, in 2022-11-23T03:54:46.3093261Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3093394Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3093586Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3093781Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3093986Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3094081Z self.run() 2022-11-23T03:54:46.3094274Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3094410Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3094762Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3094887Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3095258Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3095375Z getattr(self, test_name)() 2022-11-23T03:54:46.3095740Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3095830Z fn() 2022-11-23T03:54:46.3096209Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3096325Z test(self, **param_kwargs) 2022-11-23T03:54:46.3096689Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3096790Z return func(*args, **kwargs) 2022-11-23T03:54:46.3097031Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3097137Z self.run_subtests( 2022-11-23T03:54:46.3097498Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3097650Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3098019Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3098170Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3098555Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3098666Z output = model(*input) 2022-11-23T03:54:46.3099000Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3099135Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3099517Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3099684Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3100056Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3100169Z _lazy_init(state, module) 2022-11-23T03:54:46.3100532Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3100729Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3101082Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3101200Z return func(*args, **kwargs) 2022-11-23T03:54:46.3101572Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3101667Z p_assert( 2022-11-23T03:54:46.3102008Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3102124Z traceback.print_stack() 2022-11-23T03:54:46.3102343Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3102462Z File "", line 1, in 2022-11-23T03:54:46.3102662Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3102838Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3103036Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3103179Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3103382Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3103479Z self.run() 2022-11-23T03:54:46.3103677Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3103813Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3104162Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3104290Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3104660Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3104762Z getattr(self, test_name)() 2022-11-23T03:54:46.3105132Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3105225Z fn() 2022-11-23T03:54:46.3105597Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3105714Z test(self, **param_kwargs) 2022-11-23T03:54:46.3106078Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3106197Z return func(*args, **kwargs) 2022-11-23T03:54:46.3106436Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3106541Z self.run_subtests( 2022-11-23T03:54:46.3106902Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3107056Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3107436Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3107585Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3107971Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3108080Z output = model(*input) 2022-11-23T03:54:46.3108416Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3108549Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3108934Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3109087Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3109465Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3109579Z _lazy_init(state, module) 2022-11-23T03:54:46.3110010Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3110145Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3110490Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3110607Z return func(*args, **kwargs) 2022-11-23T03:54:46.3110993Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3111086Z p_assert( 2022-11-23T03:54:46.3111425Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3111541Z traceback.print_stack() 2022-11-23T03:54:46.3111761Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3111881Z File "", line 1, in 2022-11-23T03:54:46.3112128Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3112269Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3112459Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3112601Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3112789Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3112888Z self.run() 2022-11-23T03:54:46.3113080Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3113215Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3113563Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3113690Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3114060Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3114181Z getattr(self, test_name)() 2022-11-23T03:54:46.3114550Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3114641Z fn() 2022-11-23T03:54:46.3115015Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3115131Z test(self, **param_kwargs) 2022-11-23T03:54:46.3115497Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3115613Z return func(*args, **kwargs) 2022-11-23T03:54:46.3115856Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3115965Z self.run_subtests( 2022-11-23T03:54:46.3116322Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3116484Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3116843Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3116990Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3117374Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3117486Z output = model(*input) 2022-11-23T03:54:46.3117818Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3117950Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3118338Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3118503Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3118879Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3119061Z _lazy_init(state, module) 2022-11-23T03:54:46.3119422Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3119556Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3119900Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3120018Z return func(*args, **kwargs) 2022-11-23T03:54:46.3120403Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3120497Z p_assert( 2022-11-23T03:54:46.3120838Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3120954Z traceback.print_stack() 2022-11-23T03:54:46.3121156Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3121328Z File "", line 1, in 2022-11-23T03:54:46.3121534Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3121668Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3121860Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3122002Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3122205Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3122301Z self.run() 2022-11-23T03:54:46.3122496Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3122633Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3122989Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3123113Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3123488Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3123609Z getattr(self, test_name)() 2022-11-23T03:54:46.3123982Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3124073Z fn() 2022-11-23T03:54:46.3124446Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3124549Z test(self, **param_kwargs) 2022-11-23T03:54:46.3124913Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3125030Z return func(*args, **kwargs) 2022-11-23T03:54:46.3125271Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3125376Z self.run_subtests( 2022-11-23T03:54:46.3125737Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3125893Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3126263Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3126408Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3126790Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3126900Z output = model(*input) 2022-11-23T03:54:46.3127234Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3127366Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3127800Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3127973Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3128423Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3128536Z _lazy_init(state, module) 2022-11-23T03:54:46.3128894Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3129013Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3129360Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3129478Z return func(*args, **kwargs) 2022-11-23T03:54:46.3129861Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3129958Z p_assert( 2022-11-23T03:54:46.3130298Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3130414Z traceback.print_stack() 2022-11-23T03:54:46.3130690Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3130814Z File "", line 1, in 2022-11-23T03:54:46.3131015Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3131150Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3131342Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3131489Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3131694Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3131792Z self.run() 2022-11-23T03:54:46.3131984Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3132120Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3132457Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3132591Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3132959Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3133074Z getattr(self, test_name)() 2022-11-23T03:54:46.3133441Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3133532Z fn() 2022-11-23T03:54:46.3133904Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3134018Z test(self, **param_kwargs) 2022-11-23T03:54:46.3134382Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3134498Z return func(*args, **kwargs) 2022-11-23T03:54:46.3134737Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3134848Z self.run_subtests( 2022-11-23T03:54:46.3135209Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3135362Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3135730Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3135875Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3136258Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3136368Z output = model(*input) 2022-11-23T03:54:46.3136686Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3136820Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3137208Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3137441Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3137821Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3137936Z _lazy_init(state, module) 2022-11-23T03:54:46.3138293Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3138427Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3138771Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3138887Z return func(*args, **kwargs) 2022-11-23T03:54:46.3139272Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3139367Z p_assert( 2022-11-23T03:54:46.3139763Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3139887Z traceback.print_stack() 2022-11-23T03:54:46.3140109Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3140234Z File "", line 1, in 2022-11-23T03:54:46.3140431Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3140564Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3140741Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3140887Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3141091Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3141186Z self.run() 2022-11-23T03:54:46.3141379Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3141518Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3141876Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3142002Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3142376Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3142494Z getattr(self, test_name)() 2022-11-23T03:54:46.3142860Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3142950Z fn() 2022-11-23T03:54:46.3143322Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3143439Z test(self, **param_kwargs) 2022-11-23T03:54:46.3143804Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3143921Z return func(*args, **kwargs) 2022-11-23T03:54:46.3144167Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3144258Z self.run_subtests( 2022-11-23T03:54:46.3144617Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3144770Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3145144Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3145288Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3145671Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3145782Z output = model(*input) 2022-11-23T03:54:46.3146115Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3146248Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3146703Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3146871Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3147245Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3147357Z _lazy_init(state, module) 2022-11-23T03:54:46.3147714Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3147847Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3148194Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3148310Z return func(*args, **kwargs) 2022-11-23T03:54:46.3148695Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3148822Z p_assert( 2022-11-23T03:54:46.3149167Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3149288Z traceback.print_stack() 2022-11-23T03:54:46.3149507Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3149725Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3149944Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3150162Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3150382Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3150597Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3150815Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3151037Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3151257Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3151473Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3151689Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3151906Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3152123Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3152342Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3152559Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3152777Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3152998Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3153200Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3153418Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3153633Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3153850Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3154632Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.3155465Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.3156232Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.3157016Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.3157926Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.3158697Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.3159459Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.3160229Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.3160988Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.3161749Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.3161973Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3162199Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3162421Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3162639Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3162856Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3163071Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3163287Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3163501Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3163701Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3163964Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3164181Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3164397Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3164612Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3164828Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3165047Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3165260Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3165476Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3165692Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3165945Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3166162Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3166377Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3166589Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3166804Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3167016Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3167934Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.3168705Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.3169471Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.3170231Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.3170990Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.3171745Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.3172505Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.3173354Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.3174108Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.3174863Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.3175136Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3175365Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3175469Z dist init r=1, world=2 2022-11-23T03:54:46.3175781Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.3176090Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.3176396Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.3176702Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.3177006Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.3177306Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.3177606Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.3177905Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.3178204Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.3178509Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.3178808Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.3179105Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.3179209Z dist init r=0, world=2 2022-11-23T03:54:46.3179522Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.3179831Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.3180181Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.3180486Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.3180787Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.3181088Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.3181386Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.3181720Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.3182026Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.3182326Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.3182626Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.3182924Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.3183018Z ok (9.238s) 2022-11-23T03:54:46.3183354Z test_nested_always_wrap_model_offload_true_none_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48870 2022-11-23T03:54:46.3183565Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48871 2022-11-23T03:54:46.3183959Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.3184129Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.3184517Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.3184699Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.3184911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.3185285Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.3185456Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.3185845Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.3186025Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.3186254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.3186652Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.3187048Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.3187324Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.3187604Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.3187822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.3188096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.3188316Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3188536Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3189584Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.3189689Z warnings.warn( 2022-11-23T03:54:46.3189813Z File "", line 1, in 2022-11-23T03:54:46.3190053Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3190194Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3190388Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3190537Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3190742Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3190823Z self.run() 2022-11-23T03:54:46.3191018Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3191154Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3191505Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3191631Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3192004Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3192130Z getattr(self, test_name)() 2022-11-23T03:54:46.3192497Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3192589Z fn() 2022-11-23T03:54:46.3192962Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3193079Z test(self, **param_kwargs) 2022-11-23T03:54:46.3193452Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3193570Z return func(*args, **kwargs) 2022-11-23T03:54:46.3193813Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3193918Z self.run_subtests( 2022-11-23T03:54:46.3194281Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3194441Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3194813Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3194943Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3195325Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3195436Z output = model(*input) 2022-11-23T03:54:46.3195771Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3195907Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3196291Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3196458Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3196838Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3197019Z _lazy_init(state, module) 2022-11-23T03:54:46.3197381Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3197515Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3197860Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3197979Z return func(*args, **kwargs) 2022-11-23T03:54:46.3198368Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3198461Z p_assert( 2022-11-23T03:54:46.3198802Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3198918Z traceback.print_stack() 2022-11-23T03:54:46.3199135Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3200239Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.3200345Z warnings.warn( 2022-11-23T03:54:46.3200466Z File "", line 1, in 2022-11-23T03:54:46.3200655Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3200792Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3200985Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3201131Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3201339Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3201437Z self.run() 2022-11-23T03:54:46.3201630Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3201765Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3202114Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3202238Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3202613Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3202733Z getattr(self, test_name)() 2022-11-23T03:54:46.3203103Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3203197Z fn() 2022-11-23T03:54:46.3203575Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3203697Z test(self, **param_kwargs) 2022-11-23T03:54:46.3204048Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3204166Z return func(*args, **kwargs) 2022-11-23T03:54:46.3204409Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3204516Z self.run_subtests( 2022-11-23T03:54:46.3204874Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3205028Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3205398Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3205543Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3205929Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3206111Z output = model(*input) 2022-11-23T03:54:46.3206449Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3206583Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3206970Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3207135Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3207510Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3207625Z _lazy_init(state, module) 2022-11-23T03:54:46.3208046Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3208185Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3208591Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3208702Z return func(*args, **kwargs) 2022-11-23T03:54:46.3209098Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3209192Z p_assert( 2022-11-23T03:54:46.3209534Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3209651Z traceback.print_stack() 2022-11-23T03:54:46.3209869Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3209991Z File "", line 1, in 2022-11-23T03:54:46.3210189Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3210323Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3210515Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3210663Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3210866Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3210960Z self.run() 2022-11-23T03:54:46.3211158Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3211292Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3211638Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3211768Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3212125Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3212240Z getattr(self, test_name)() 2022-11-23T03:54:46.3212607Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3212698Z fn() 2022-11-23T03:54:46.3213078Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3213195Z test(self, **param_kwargs) 2022-11-23T03:54:46.3213560Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3213677Z return func(*args, **kwargs) 2022-11-23T03:54:46.3213919Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3214024Z self.run_subtests( 2022-11-23T03:54:46.3214385Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3214540Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3214910Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3215054Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3215513Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3215625Z output = model(*input) 2022-11-23T03:54:46.3215959Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3216094Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3216464Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3216629Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3217003Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3217118Z _lazy_init(state, module) 2022-11-23T03:54:46.3217479Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3217663Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3218019Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3218135Z return func(*args, **kwargs) 2022-11-23T03:54:46.3218522Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3218616Z p_assert( 2022-11-23T03:54:46.3218956Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3219072Z traceback.print_stack() 2022-11-23T03:54:46.3219292Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3219415Z File "", line 1, in 2022-11-23T03:54:46.3219613Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3219751Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3219949Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3220077Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3220281Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3220378Z self.run() 2022-11-23T03:54:46.3220572Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3220708Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3221059Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3221185Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3221554Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3221670Z getattr(self, test_name)() 2022-11-23T03:54:46.3222039Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3222132Z fn() 2022-11-23T03:54:46.3222507Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3222622Z test(self, **param_kwargs) 2022-11-23T03:54:46.3222986Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3223101Z return func(*args, **kwargs) 2022-11-23T03:54:46.3223346Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3223451Z self.run_subtests( 2022-11-23T03:54:46.3223810Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3223949Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3224324Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3224529Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3224918Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3225032Z output = model(*input) 2022-11-23T03:54:46.3225369Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3225502Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3225890Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3226056Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3226436Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3226550Z _lazy_init(state, module) 2022-11-23T03:54:46.3226956Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3227096Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3227444Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3227560Z return func(*args, **kwargs) 2022-11-23T03:54:46.3227946Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3228039Z p_assert( 2022-11-23T03:54:46.3228381Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3228484Z traceback.print_stack() 2022-11-23T03:54:46.3228708Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3228827Z File "", line 1, in 2022-11-23T03:54:46.3229031Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3229170Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3229360Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3229502Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3229709Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3229805Z self.run() 2022-11-23T03:54:46.3229998Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3230136Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3230485Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3230614Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3230986Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3231112Z getattr(self, test_name)() 2022-11-23T03:54:46.3231481Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3231571Z fn() 2022-11-23T03:54:46.3231930Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3232048Z test(self, **param_kwargs) 2022-11-23T03:54:46.3232414Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3232532Z return func(*args, **kwargs) 2022-11-23T03:54:46.3232772Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3232877Z self.run_subtests( 2022-11-23T03:54:46.3233240Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3233396Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3233832Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3233976Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3234360Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3234472Z output = model(*input) 2022-11-23T03:54:46.3234805Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3234939Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3235321Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3235486Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3235903Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3236024Z _lazy_init(state, module) 2022-11-23T03:54:46.3236370Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3236504Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3236848Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3236966Z return func(*args, **kwargs) 2022-11-23T03:54:46.3237352Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3237447Z p_assert( 2022-11-23T03:54:46.3237786Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3237904Z traceback.print_stack() 2022-11-23T03:54:46.3238123Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3238247Z File "", line 1, in 2022-11-23T03:54:46.3238447Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3238577Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3238767Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3238912Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3239115Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3239211Z self.run() 2022-11-23T03:54:46.3239403Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3239526Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3239873Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3239998Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3240370Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3240490Z getattr(self, test_name)() 2022-11-23T03:54:46.3240857Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3240951Z fn() 2022-11-23T03:54:46.3241324Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3241441Z test(self, **param_kwargs) 2022-11-23T03:54:46.3241806Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3241923Z return func(*args, **kwargs) 2022-11-23T03:54:46.3242166Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3242270Z self.run_subtests( 2022-11-23T03:54:46.3242631Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3242860Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3243231Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3243376Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3243759Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3243855Z output = model(*input) 2022-11-23T03:54:46.3244192Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3244324Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3244708Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3244876Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3245300Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3245420Z _lazy_init(state, module) 2022-11-23T03:54:46.3245783Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3245915Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3246260Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3246376Z return func(*args, **kwargs) 2022-11-23T03:54:46.3246760Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3246855Z p_assert( 2022-11-23T03:54:46.3247196Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3247313Z traceback.print_stack() 2022-11-23T03:54:46.3247537Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3247657Z File "", line 1, in 2022-11-23T03:54:46.3247903Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3248023Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3248215Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3248361Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3248565Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3248659Z self.run() 2022-11-23T03:54:46.3248856Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3248990Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3249338Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3249470Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3249841Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3249957Z getattr(self, test_name)() 2022-11-23T03:54:46.3250323Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3250414Z fn() 2022-11-23T03:54:46.3250786Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3250903Z test(self, **param_kwargs) 2022-11-23T03:54:46.3251268Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3251385Z return func(*args, **kwargs) 2022-11-23T03:54:46.3251613Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3251796Z self.run_subtests( 2022-11-23T03:54:46.3252158Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3252319Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3252688Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3252833Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3253215Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3253325Z output = model(*input) 2022-11-23T03:54:46.3253661Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3253794Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3254228Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3254399Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3254781Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3254896Z _lazy_init(state, module) 2022-11-23T03:54:46.3255254Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3255386Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3255730Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3255849Z return func(*args, **kwargs) 2022-11-23T03:54:46.3256219Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3256315Z p_assert( 2022-11-23T03:54:46.3256659Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3256777Z traceback.print_stack() 2022-11-23T03:54:46.3257000Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3257120Z File "", line 1, in 2022-11-23T03:54:46.3257319Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3257453Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3257646Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3257789Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3257991Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3258088Z self.run() 2022-11-23T03:54:46.3258282Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3258421Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3258775Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3258899Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3259269Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3259371Z getattr(self, test_name)() 2022-11-23T03:54:46.3259736Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3259827Z fn() 2022-11-23T03:54:46.3260198Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3260314Z test(self, **param_kwargs) 2022-11-23T03:54:46.3260679Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3260800Z return func(*args, **kwargs) 2022-11-23T03:54:46.3261046Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3261218Z self.run_subtests( 2022-11-23T03:54:46.3261581Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3261733Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3262104Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3262249Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3262630Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3262741Z output = model(*input) 2022-11-23T03:54:46.3263074Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3263208Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3263639Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3263795Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3264174Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3264287Z _lazy_init(state, module) 2022-11-23T03:54:46.3264648Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3264781Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3265130Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3265246Z return func(*args, **kwargs) 2022-11-23T03:54:46.3265632Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3265730Z p_assert( 2022-11-23T03:54:46.3266071Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3266186Z traceback.print_stack() 2022-11-23T03:54:46.3266410Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3266532Z File "", line 1, in 2022-11-23T03:54:46.3266733Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3266866Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3267056Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3267199Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3267405Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3267487Z self.run() 2022-11-23T03:54:46.3267681Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3267825Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3268176Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3268305Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3268674Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3268791Z getattr(self, test_name)() 2022-11-23T03:54:46.3269158Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3269249Z fn() 2022-11-23T03:54:46.3269626Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3269741Z test(self, **param_kwargs) 2022-11-23T03:54:46.3270109Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3270294Z return func(*args, **kwargs) 2022-11-23T03:54:46.3270541Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3270646Z self.run_subtests( 2022-11-23T03:54:46.3271009Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3271161Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3271518Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3271665Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3272048Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3272160Z output = model(*input) 2022-11-23T03:54:46.3272554Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3272694Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3273084Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3273251Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3273630Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3273745Z _lazy_init(state, module) 2022-11-23T03:54:46.3274104Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3274238Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3274581Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3274698Z return func(*args, **kwargs) 2022-11-23T03:54:46.3275087Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3275184Z p_assert( 2022-11-23T03:54:46.3275525Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3275643Z traceback.print_stack() 2022-11-23T03:54:46.3275849Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3275968Z File "", line 1, in 2022-11-23T03:54:46.3276167Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3276302Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3276493Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3276635Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3276838Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3276940Z self.run() 2022-11-23T03:54:46.3277131Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3277272Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3277620Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3277746Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3278115Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3278229Z getattr(self, test_name)() 2022-11-23T03:54:46.3278597Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3278689Z fn() 2022-11-23T03:54:46.3279061Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3279163Z test(self, **param_kwargs) 2022-11-23T03:54:46.3279602Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3279720Z return func(*args, **kwargs) 2022-11-23T03:54:46.3279963Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3280069Z self.run_subtests( 2022-11-23T03:54:46.3280433Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3280586Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3280957Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3281106Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3281489Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3281650Z output = model(*input) 2022-11-23T03:54:46.3281992Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3282125Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3282510Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3282677Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3283053Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3283166Z _lazy_init(state, module) 2022-11-23T03:54:46.3283527Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3283661Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3283993Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3284116Z return func(*args, **kwargs) 2022-11-23T03:54:46.3284503Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3284598Z p_assert( 2022-11-23T03:54:46.3284940Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3285056Z traceback.print_stack() 2022-11-23T03:54:46.3285275Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3285399Z File "", line 1, in 2022-11-23T03:54:46.3285598Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3285731Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3285925Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3286067Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3286276Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3286371Z self.run() 2022-11-23T03:54:46.3286566Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3286702Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3287034Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3287159Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3287530Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3287650Z getattr(self, test_name)() 2022-11-23T03:54:46.3288144Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3288235Z fn() 2022-11-23T03:54:46.3288611Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3288801Z test(self, **param_kwargs) 2022-11-23T03:54:46.3289171Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3289286Z return func(*args, **kwargs) 2022-11-23T03:54:46.3289527Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3289632Z self.run_subtests( 2022-11-23T03:54:46.3289995Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3290150Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3290519Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3290662Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3291104Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3291225Z output = model(*input) 2022-11-23T03:54:46.3291549Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3291683Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3292066Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3292231Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3292605Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3292719Z _lazy_init(state, module) 2022-11-23T03:54:46.3293079Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3293216Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3293566Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3293685Z return func(*args, **kwargs) 2022-11-23T03:54:46.3294074Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3294167Z p_assert( 2022-11-23T03:54:46.3294512Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3294628Z traceback.print_stack() 2022-11-23T03:54:46.3294846Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3294964Z File "", line 1, in 2022-11-23T03:54:46.3295164Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3295297Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3295479Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3295625Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3295826Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3295922Z self.run() 2022-11-23T03:54:46.3296115Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3296251Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3296601Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3296728Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3297101Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3297217Z getattr(self, test_name)() 2022-11-23T03:54:46.3297586Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3297741Z fn() 2022-11-23T03:54:46.3298124Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3298240Z test(self, **param_kwargs) 2022-11-23T03:54:46.3298607Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3298725Z return func(*args, **kwargs) 2022-11-23T03:54:46.3298965Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3299057Z self.run_subtests( 2022-11-23T03:54:46.3299415Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3299568Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3299939Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3300131Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3300522Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3300632Z output = model(*input) 2022-11-23T03:54:46.3300969Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3301106Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3301488Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3301654Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3302029Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3302144Z _lazy_init(state, module) 2022-11-23T03:54:46.3302507Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3302649Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3302994Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3303112Z return func(*args, **kwargs) 2022-11-23T03:54:46.3303498Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3303592Z p_assert( 2022-11-23T03:54:46.3303918Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3304033Z traceback.print_stack() 2022-11-23T03:54:46.3304257Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3304377Z File "", line 1, in 2022-11-23T03:54:46.3304579Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3304719Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3304911Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3305053Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3305257Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3305352Z self.run() 2022-11-23T03:54:46.3305544Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3305686Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3306033Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3306157Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3306527Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3306643Z getattr(self, test_name)() 2022-11-23T03:54:46.3306999Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3307154Z fn() 2022-11-23T03:54:46.3307531Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3307647Z test(self, **param_kwargs) 2022-11-23T03:54:46.3308016Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3308133Z return func(*args, **kwargs) 2022-11-23T03:54:46.3308370Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3308476Z self.run_subtests( 2022-11-23T03:54:46.3308837Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3308991Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3309405Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3309557Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3309943Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3310054Z output = model(*input) 2022-11-23T03:54:46.3310390Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3310526Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3310913Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3311077Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3311456Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3311556Z _lazy_init(state, module) 2022-11-23T03:54:46.3311916Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3312049Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3312396Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3312514Z return func(*args, **kwargs) 2022-11-23T03:54:46.3312900Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3312994Z p_assert( 2022-11-23T03:54:46.3313337Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3313453Z traceback.print_stack() 2022-11-23T03:54:46.3313678Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3313796Z File "", line 1, in 2022-11-23T03:54:46.3314003Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3314137Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3314329Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3314471Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3314672Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3314767Z self.run() 2022-11-23T03:54:46.3314948Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3315085Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3315431Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3315558Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3315928Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3316120Z getattr(self, test_name)() 2022-11-23T03:54:46.3316489Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3316580Z fn() 2022-11-23T03:54:46.3316952Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3317067Z test(self, **param_kwargs) 2022-11-23T03:54:46.3317433Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3317550Z return func(*args, **kwargs) 2022-11-23T03:54:46.3317785Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3317887Z self.run_subtests( 2022-11-23T03:54:46.3318240Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3318435Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3318807Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3318937Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3319318Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3319425Z output = model(*input) 2022-11-23T03:54:46.3319754Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3319882Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3320259Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3320420Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3320792Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3320903Z _lazy_init(state, module) 2022-11-23T03:54:46.3321255Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3321383Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3321722Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3321832Z return func(*args, **kwargs) 2022-11-23T03:54:46.3322213Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3322302Z p_assert( 2022-11-23T03:54:46.3322636Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3322746Z traceback.print_stack() 2022-11-23T03:54:46.3322961Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3323071Z File "", line 1, in 2022-11-23T03:54:46.3323265Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3323393Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3323579Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3323716Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3323911Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3324002Z self.run() 2022-11-23T03:54:46.3324190Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3324321Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3324660Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3324778Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3325213Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3325325Z getattr(self, test_name)() 2022-11-23T03:54:46.3325686Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3325771Z fn() 2022-11-23T03:54:46.3326137Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3326247Z test(self, **param_kwargs) 2022-11-23T03:54:46.3326598Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3326708Z return func(*args, **kwargs) 2022-11-23T03:54:46.3326942Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3327042Z self.run_subtests( 2022-11-23T03:54:46.3327441Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3327595Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3328014Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3328152Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3328530Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3328634Z output = model(*input) 2022-11-23T03:54:46.3328962Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3329090Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3329472Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3329635Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3330006Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3330114Z _lazy_init(state, module) 2022-11-23T03:54:46.3330467Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3330594Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3330933Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3331037Z return func(*args, **kwargs) 2022-11-23T03:54:46.3331416Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3331504Z p_assert( 2022-11-23T03:54:46.3331838Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3331955Z traceback.print_stack() 2022-11-23T03:54:46.3332168Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3332283Z File "", line 1, in 2022-11-23T03:54:46.3332474Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3332601Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3332787Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3332924Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3333121Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3333211Z self.run() 2022-11-23T03:54:46.3333401Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3333531Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3333875Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3334059Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3334428Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3334538Z getattr(self, test_name)() 2022-11-23T03:54:46.3334901Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3334986Z fn() 2022-11-23T03:54:46.3335353Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3335463Z test(self, **param_kwargs) 2022-11-23T03:54:46.3335824Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3335937Z return func(*args, **kwargs) 2022-11-23T03:54:46.3336174Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3336328Z self.run_subtests( 2022-11-23T03:54:46.3336688Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3336837Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3337202Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3337339Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3337717Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3337823Z output = model(*input) 2022-11-23T03:54:46.3338152Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3338278Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3338650Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3338815Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3339183Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3339291Z _lazy_init(state, module) 2022-11-23T03:54:46.3339643Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3339770Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3340109Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3340220Z return func(*args, **kwargs) 2022-11-23T03:54:46.3340600Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3340689Z p_assert( 2022-11-23T03:54:46.3341026Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3341140Z traceback.print_stack() 2022-11-23T03:54:46.3341352Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3341468Z File "", line 1, in 2022-11-23T03:54:46.3341660Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3341788Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3341974Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3342102Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3342300Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3342390Z self.run() 2022-11-23T03:54:46.3342577Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3342706Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3343133Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3343254Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3343618Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3343728Z getattr(self, test_name)() 2022-11-23T03:54:46.3344088Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3344174Z fn() 2022-11-23T03:54:46.3344540Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3344650Z test(self, **param_kwargs) 2022-11-23T03:54:46.3345010Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3345120Z return func(*args, **kwargs) 2022-11-23T03:54:46.3345426Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3345528Z self.run_subtests( 2022-11-23T03:54:46.3345876Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3346027Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3346392Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3346530Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3346908Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3347013Z output = model(*input) 2022-11-23T03:54:46.3347340Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3347476Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3347855Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3348017Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3348387Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3348495Z _lazy_init(state, module) 2022-11-23T03:54:46.3348848Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3348975Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3349313Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3349423Z return func(*args, **kwargs) 2022-11-23T03:54:46.3349805Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3349896Z p_assert( 2022-11-23T03:54:46.3350229Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3350332Z traceback.print_stack() 2022-11-23T03:54:46.3350547Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3350661Z File "", line 1, in 2022-11-23T03:54:46.3350856Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3350984Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3351169Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3351305Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3351500Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3351590Z self.run() 2022-11-23T03:54:46.3351779Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3351977Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3352326Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3352446Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3352811Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3352922Z getattr(self, test_name)() 2022-11-23T03:54:46.3353283Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3353359Z fn() 2022-11-23T03:54:46.3353726Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3353836Z test(self, **param_kwargs) 2022-11-23T03:54:46.3354242Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3354362Z return func(*args, **kwargs) 2022-11-23T03:54:46.3354597Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3354695Z self.run_subtests( 2022-11-23T03:54:46.3355053Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3355203Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3355568Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3355706Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3356084Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3356189Z output = model(*input) 2022-11-23T03:54:46.3356521Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3356652Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3357028Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3357189Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3357559Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3357667Z _lazy_init(state, module) 2022-11-23T03:54:46.3358012Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3358140Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3358478Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3358589Z return func(*args, **kwargs) 2022-11-23T03:54:46.3358975Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3359065Z p_assert( 2022-11-23T03:54:46.3359403Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3359513Z traceback.print_stack() 2022-11-23T03:54:46.3359724Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3359839Z File "", line 1, in 2022-11-23T03:54:46.3360031Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3360158Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3360344Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3360480Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3360679Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3360838Z self.run() 2022-11-23T03:54:46.3361027Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3361149Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3361493Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3361615Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3361980Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3362092Z getattr(self, test_name)() 2022-11-23T03:54:46.3362455Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3362540Z fn() 2022-11-23T03:54:46.3362910Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3363024Z test(self, **param_kwargs) 2022-11-23T03:54:46.3363429Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3363543Z return func(*args, **kwargs) 2022-11-23T03:54:46.3363779Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3363876Z self.run_subtests( 2022-11-23T03:54:46.3364232Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3364381Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3364746Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3364888Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3365266Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3365370Z output = model(*input) 2022-11-23T03:54:46.3365702Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3365829Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3366208Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3366368Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3366737Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3366846Z _lazy_init(state, module) 2022-11-23T03:54:46.3367197Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3367325Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3367669Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3367825Z return func(*args, **kwargs) 2022-11-23T03:54:46.3368207Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3368295Z p_assert( 2022-11-23T03:54:46.3368630Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3368741Z traceback.print_stack() 2022-11-23T03:54:46.3368956Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3369070Z File "", line 1, in 2022-11-23T03:54:46.3369262Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3369382Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3369569Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3369710Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3369982Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3370072Z self.run() 2022-11-23T03:54:46.3370259Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3370389Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3370731Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3370855Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3371216Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3378325Z getattr(self, test_name)() 2022-11-23T03:54:46.3378749Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3378838Z fn() 2022-11-23T03:54:46.3379353Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3379472Z test(self, **param_kwargs) 2022-11-23T03:54:46.3379841Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3379953Z return func(*args, **kwargs) 2022-11-23T03:54:46.3380191Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3380290Z self.run_subtests( 2022-11-23T03:54:46.3380644Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3380796Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3381166Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3381306Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3381699Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3381804Z output = model(*input) 2022-11-23T03:54:46.3382137Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3382257Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3382636Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3382800Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3383169Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3383282Z _lazy_init(state, module) 2022-11-23T03:54:46.3383635Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3383773Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3384110Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3384226Z return func(*args, **kwargs) 2022-11-23T03:54:46.3384609Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3384698Z p_assert( 2022-11-23T03:54:46.3385035Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3385146Z traceback.print_stack() 2022-11-23T03:54:46.3385362Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3385478Z File "", line 1, in 2022-11-23T03:54:46.3385677Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3385809Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3386059Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3386196Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3386395Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3386485Z self.run() 2022-11-23T03:54:46.3386679Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3386810Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3387157Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3387283Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3387648Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3387765Z getattr(self, test_name)() 2022-11-23T03:54:46.3388129Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3388266Z fn() 2022-11-23T03:54:46.3388645Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3388760Z test(self, **param_kwargs) 2022-11-23T03:54:46.3389123Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3389237Z return func(*args, **kwargs) 2022-11-23T03:54:46.3389480Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3389570Z self.run_subtests( 2022-11-23T03:54:46.3389927Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3390080Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3390448Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3390602Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3390986Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3391097Z output = model(*input) 2022-11-23T03:54:46.3391432Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3391565Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3391946Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3392115Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3392487Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3392600Z _lazy_init(state, module) 2022-11-23T03:54:46.3392961Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3393097Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3393442Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3393558Z return func(*args, **kwargs) 2022-11-23T03:54:46.3393940Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3394033Z p_assert( 2022-11-23T03:54:46.3394359Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3394475Z traceback.print_stack() 2022-11-23T03:54:46.3394692Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3394812Z File "", line 1, in 2022-11-23T03:54:46.3395012Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3395210Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3395400Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3395542Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3395745Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3395841Z self.run() 2022-11-23T03:54:46.3396032Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3396167Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3396516Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3396640Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3397009Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3397125Z getattr(self, test_name)() 2022-11-23T03:54:46.3397538Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3397617Z fn() 2022-11-23T03:54:46.3397990Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3398106Z test(self, **param_kwargs) 2022-11-23T03:54:46.3398467Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3398584Z return func(*args, **kwargs) 2022-11-23T03:54:46.3398822Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3398927Z self.run_subtests( 2022-11-23T03:54:46.3399283Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3399436Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3399810Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3399953Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3400332Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3400448Z output = model(*input) 2022-11-23T03:54:46.3400779Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3400914Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3401295Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3401463Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3401836Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3401941Z _lazy_init(state, module) 2022-11-23T03:54:46.3402300Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3402434Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3402777Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3402893Z return func(*args, **kwargs) 2022-11-23T03:54:46.3403278Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3403374Z p_assert( 2022-11-23T03:54:46.3403712Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3403829Z traceback.print_stack() 2022-11-23T03:54:46.3404050Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3404167Z File "", line 1, in 2022-11-23T03:54:46.3404429Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3404562Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3404752Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3404893Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3405095Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3405191Z self.run() 2022-11-23T03:54:46.3405367Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3405503Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3405852Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3405977Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3406387Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3406509Z getattr(self, test_name)() 2022-11-23T03:54:46.3406878Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3406966Z fn() 2022-11-23T03:54:46.3407335Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3407451Z test(self, **param_kwargs) 2022-11-23T03:54:46.3408211Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3408434Z return func(*args, **kwargs) 2022-11-23T03:54:46.3408785Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3408890Z self.run_subtests( 2022-11-23T03:54:46.3409255Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3409416Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3409785Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3409926Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3410296Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3410405Z output = model(*input) 2022-11-23T03:54:46.3410737Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3410869Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3411247Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3411409Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3411786Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3411903Z _lazy_init(state, module) 2022-11-23T03:54:46.3412261Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3412743Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3413372Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3413831Z return func(*args, **kwargs) 2022-11-23T03:54:46.3414481Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3414945Z p_assert( 2022-11-23T03:54:46.3415523Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3415978Z traceback.print_stack() 2022-11-23T03:54:46.3416440Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3417003Z File "", line 1, in 2022-11-23T03:54:46.3417438Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3417851Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3418277Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3418718Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3419166Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3419551Z self.run() 2022-11-23T03:54:46.3419949Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3420372Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3420996Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3421448Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3422179Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3422652Z getattr(self, test_name)() 2022-11-23T03:54:46.3423279Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3423710Z fn() 2022-11-23T03:54:46.3424296Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3424762Z test(self, **param_kwargs) 2022-11-23T03:54:46.3425384Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3425846Z return func(*args, **kwargs) 2022-11-23T03:54:46.3426324Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3426757Z self.run_subtests( 2022-11-23T03:54:46.3427319Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3427729Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3428286Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3428704Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3429267Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3429648Z output = model(*input) 2022-11-23T03:54:46.3430132Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3430508Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3431057Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3431506Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3432063Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3432444Z _lazy_init(state, module) 2022-11-23T03:54:46.3432953Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3433361Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3433874Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3434241Z return func(*args, **kwargs) 2022-11-23T03:54:46.3434766Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3435134Z p_assert( 2022-11-23T03:54:46.3435605Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3436039Z traceback.print_stack() 2022-11-23T03:54:46.3436415Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3436871Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3437334Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3437781Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3438245Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3438702Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3439156Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3439619Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3440127Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3440584Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3441043Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3441501Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3441948Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3442409Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3442863Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3443319Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3443777Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3444240Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3444699Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3445136Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3445592Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3446053Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3446510Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3446963Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3447419Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3447928Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3448398Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3448859Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3449303Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3449753Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3450208Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3450664Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3451119Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3451571Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3452025Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3452531Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3452989Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3453447Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3453902Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3454367Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3454825Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3455283Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3455727Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3456243Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3456709Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3457164Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3457616Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3457958Z dist init r=0, world=2 2022-11-23T03:54:46.3458420Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.3459059Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.3459702Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.3460323Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.3461001Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.3461754Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.3462512Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.3463263Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.3464015Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.3464764Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.3465530Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.3466283Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.3466794Z dist init r=1, world=2 2022-11-23T03:54:46.3467327Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.3468149Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.3468904Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.3469652Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.3470397Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.3471138Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.3471933Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.3472687Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.3473425Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.3474167Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.3474911Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.3475659Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.3476147Z ok (9.941s) 2022-11-23T03:54:46.3476714Z test_nested_always_wrap_model_offload_true_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49023 2022-11-23T03:54:46.3477396Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49024 2022-11-23T03:54:46.3478056Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.3478490Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.3479066Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.3479520Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.3479960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.3480581Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.3481016Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.3481591Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.3482047Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.3482479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.3483125Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.3483805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.3484356Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.3485045Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.3485484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.3485939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.3486399Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3486857Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3488548Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.3489327Z warnings.warn( 2022-11-23T03:54:46.3489586Z File "", line 1, in 2022-11-23T03:54:46.3489935Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3490292Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3490643Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3491000Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3491371Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3491688Z self.run() 2022-11-23T03:54:46.3491997Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3492350Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3492878Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3493256Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3493784Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3494160Z getattr(self, test_name)() 2022-11-23T03:54:46.3494676Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3495022Z fn() 2022-11-23T03:54:46.3495512Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3495887Z test(self, **param_kwargs) 2022-11-23T03:54:46.3496401Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3496777Z return func(*args, **kwargs) 2022-11-23T03:54:46.3497167Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3497517Z self.run_subtests( 2022-11-23T03:54:46.3498022Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3498428Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3498974Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3499379Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3500060Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3500513Z output = model(*input) 2022-11-23T03:54:46.3501078Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3501527Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3502185Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3502803Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3503489Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3503945Z _lazy_init(state, module) 2022-11-23T03:54:46.3504562Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3505026Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3505640Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3506092Z return func(*args, **kwargs) 2022-11-23T03:54:46.3506742Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3507180Z p_assert( 2022-11-23T03:54:46.3507818Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3508271Z traceback.print_stack() 2022-11-23T03:54:46.3508702Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3510266Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.3511177Z warnings.warn( 2022-11-23T03:54:46.3511484Z File "", line 1, in 2022-11-23T03:54:46.3511921Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3512355Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3512792Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3513212Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3513675Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3514066Z self.run() 2022-11-23T03:54:46.3514461Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3514895Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3515511Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3515954Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3516577Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3517034Z getattr(self, test_name)() 2022-11-23T03:54:46.3517565Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3517924Z fn() 2022-11-23T03:54:46.3518416Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3518802Z test(self, **param_kwargs) 2022-11-23T03:54:46.3519308Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3519688Z return func(*args, **kwargs) 2022-11-23T03:54:46.3520084Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3520450Z self.run_subtests( 2022-11-23T03:54:46.3520951Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3521367Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3521924Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3522388Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3522947Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3523333Z output = model(*input) 2022-11-23T03:54:46.3523819Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3524201Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3524755Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3525195Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3525750Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3526134Z _lazy_init(state, module) 2022-11-23T03:54:46.3526699Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3527105Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3527628Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3528347Z return func(*args, **kwargs) 2022-11-23T03:54:46.3528899Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3529258Z p_assert( 2022-11-23T03:54:46.3529733Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3530103Z traceback.print_stack() 2022-11-23T03:54:46.3530476Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3530839Z File "", line 1, in 2022-11-23T03:54:46.3531196Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3531553Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3531910Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3532268Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3532649Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3532977Z self.run() 2022-11-23T03:54:46.3533303Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3533649Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3534163Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3534540Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3535071Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3535455Z getattr(self, test_name)() 2022-11-23T03:54:46.3535980Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3536338Z fn() 2022-11-23T03:54:46.3536827Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3537210Z test(self, **param_kwargs) 2022-11-23T03:54:46.3537726Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3538112Z return func(*args, **kwargs) 2022-11-23T03:54:46.3538510Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3538923Z self.run_subtests( 2022-11-23T03:54:46.3539514Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3540015Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3540791Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3541284Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3541956Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3542414Z output = model(*input) 2022-11-23T03:54:46.3542996Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3543433Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3544095Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3544621Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3545302Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3545837Z _lazy_init(state, module) 2022-11-23T03:54:46.3546472Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3546946Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3547559Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3548006Z return func(*args, **kwargs) 2022-11-23T03:54:46.3548657Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3549105Z p_assert( 2022-11-23T03:54:46.3549676Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3550128Z traceback.print_stack() 2022-11-23T03:54:46.3550563Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3551005Z File "", line 1, in 2022-11-23T03:54:46.3551445Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3551880Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3552310Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3552735Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3553191Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3553571Z self.run() 2022-11-23T03:54:46.3553961Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3554392Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3555013Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3555468Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3556109Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3556555Z getattr(self, test_name)() 2022-11-23T03:54:46.3557185Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3557613Z fn() 2022-11-23T03:54:46.3558132Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3558516Z test(self, **param_kwargs) 2022-11-23T03:54:46.3559036Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3559416Z return func(*args, **kwargs) 2022-11-23T03:54:46.3559800Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3560165Z self.run_subtests( 2022-11-23T03:54:46.3560671Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3561170Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3561727Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3562136Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3562690Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3563064Z output = model(*input) 2022-11-23T03:54:46.3563552Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3563931Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3564481Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3564925Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3565549Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3565938Z _lazy_init(state, module) 2022-11-23T03:54:46.3566441Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3566836Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3567355Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3567782Z return func(*args, **kwargs) 2022-11-23T03:54:46.3568334Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3568709Z p_assert( 2022-11-23T03:54:46.3569171Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3569543Z traceback.print_stack() 2022-11-23T03:54:46.3569925Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3570299Z File "", line 1, in 2022-11-23T03:54:46.3570657Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3571014Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3571361Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3571722Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3572103Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3572428Z self.run() 2022-11-23T03:54:46.3572755Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3573114Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3573627Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3573991Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3574529Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3574918Z getattr(self, test_name)() 2022-11-23T03:54:46.3575441Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3575798Z fn() 2022-11-23T03:54:46.3576295Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3576667Z test(self, **param_kwargs) 2022-11-23T03:54:46.3577183Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3577566Z return func(*args, **kwargs) 2022-11-23T03:54:46.3577962Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3578327Z self.run_subtests( 2022-11-23T03:54:46.3578906Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3579322Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3579863Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3580276Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3580833Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3581215Z output = model(*input) 2022-11-23T03:54:46.3581702Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3582081Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3582633Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3583119Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3583695Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3584077Z _lazy_init(state, module) 2022-11-23T03:54:46.3584590Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3584982Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3585497Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3585868Z return func(*args, **kwargs) 2022-11-23T03:54:46.3586396Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3586766Z p_assert( 2022-11-23T03:54:46.3587238Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3587615Z traceback.print_stack() 2022-11-23T03:54:46.3587995Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3588356Z File "", line 1, in 2022-11-23T03:54:46.3588705Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3589063Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3589427Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3589786Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3590165Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3590490Z self.run() 2022-11-23T03:54:46.3590804Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3591161Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3591682Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3592062Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3592591Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3592976Z getattr(self, test_name)() 2022-11-23T03:54:46.3593500Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3593846Z fn() 2022-11-23T03:54:46.3594342Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3594728Z test(self, **param_kwargs) 2022-11-23T03:54:46.3595245Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3595623Z return func(*args, **kwargs) 2022-11-23T03:54:46.3596023Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3596445Z self.run_subtests( 2022-11-23T03:54:46.3596956Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3597371Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3597923Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3598335Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3598896Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3599279Z output = model(*input) 2022-11-23T03:54:46.3599752Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3600130Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3600731Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3601181Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3601751Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3602133Z _lazy_init(state, module) 2022-11-23T03:54:46.3602646Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3603030Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3603549Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3603920Z return func(*args, **kwargs) 2022-11-23T03:54:46.3604465Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3604838Z p_assert( 2022-11-23T03:54:46.3605315Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3605685Z traceback.print_stack() 2022-11-23T03:54:46.3606050Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3606450Z File "", line 1, in 2022-11-23T03:54:46.3606885Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3607322Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3607921Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3608502Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3608940Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3609328Z self.run() 2022-11-23T03:54:46.3609722Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3610158Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3610799Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3611250Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3611872Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3612334Z getattr(self, test_name)() 2022-11-23T03:54:46.3612961Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3613382Z fn() 2022-11-23T03:54:46.3613985Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3614445Z test(self, **param_kwargs) 2022-11-23T03:54:46.3615074Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3615623Z return func(*args, **kwargs) 2022-11-23T03:54:46.3616103Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3616542Z self.run_subtests( 2022-11-23T03:54:46.3617157Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3617661Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3618225Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3618635Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3619182Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3619564Z output = model(*input) 2022-11-23T03:54:46.3620048Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3620490Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3621051Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3621490Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3622044Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3622429Z _lazy_init(state, module) 2022-11-23T03:54:46.3622941Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3623336Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3623853Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3624221Z return func(*args, **kwargs) 2022-11-23T03:54:46.3624767Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3625130Z p_assert( 2022-11-23T03:54:46.3625604Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3625973Z traceback.print_stack() 2022-11-23T03:54:46.3626346Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3626713Z File "", line 1, in 2022-11-23T03:54:46.3627073Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3627435Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3627784Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3628147Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3628526Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3628850Z self.run() 2022-11-23T03:54:46.3629187Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3629545Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3630049Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3630427Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3630956Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3631340Z getattr(self, test_name)() 2022-11-23T03:54:46.3631857Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3632215Z fn() 2022-11-23T03:54:46.3632698Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3633085Z test(self, **param_kwargs) 2022-11-23T03:54:46.3633604Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3634058Z return func(*args, **kwargs) 2022-11-23T03:54:46.3634454Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3634818Z self.run_subtests( 2022-11-23T03:54:46.3635326Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3635728Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3636283Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3636693Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3637249Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3637634Z output = model(*input) 2022-11-23T03:54:46.3638174Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3638560Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3639103Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3639546Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3640110Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3640493Z _lazy_init(state, module) 2022-11-23T03:54:46.3641004Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3641398Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3641910Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3642274Z return func(*args, **kwargs) 2022-11-23T03:54:46.3642816Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3643189Z p_assert( 2022-11-23T03:54:46.3643663Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3644034Z traceback.print_stack() 2022-11-23T03:54:46.3644410Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3644764Z File "", line 1, in 2022-11-23T03:54:46.3645123Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3645482Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3645840Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3646200Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3646581Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3646909Z self.run() 2022-11-23T03:54:46.3647225Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3647579Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3648156Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3648531Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3649063Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3649447Z getattr(self, test_name)() 2022-11-23T03:54:46.3649957Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3650313Z fn() 2022-11-23T03:54:46.3650807Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3651305Z test(self, **param_kwargs) 2022-11-23T03:54:46.3651831Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3651946Z return func(*args, **kwargs) 2022-11-23T03:54:46.3652187Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3652289Z self.run_subtests( 2022-11-23T03:54:46.3652646Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3652798Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3653167Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3653299Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3653734Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3653851Z output = model(*input) 2022-11-23T03:54:46.3654188Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3654320Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3654703Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3654868Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3655243Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3655355Z _lazy_init(state, module) 2022-11-23T03:54:46.3655712Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3655848Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3656196Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3656315Z return func(*args, **kwargs) 2022-11-23T03:54:46.3656700Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3656792Z p_assert( 2022-11-23T03:54:46.3657133Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3657246Z traceback.print_stack() 2022-11-23T03:54:46.3657465Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3657572Z File "", line 1, in 2022-11-23T03:54:46.3657769Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3657901Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3658093Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3658240Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3658442Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3658536Z self.run() 2022-11-23T03:54:46.3658731Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3658865Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3659214Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3659338Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3659707Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3659821Z getattr(self, test_name)() 2022-11-23T03:54:46.3660185Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3660276Z fn() 2022-11-23T03:54:46.3660705Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3660819Z test(self, **param_kwargs) 2022-11-23T03:54:46.3661173Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3661292Z return func(*args, **kwargs) 2022-11-23T03:54:46.3661534Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3661638Z self.run_subtests( 2022-11-23T03:54:46.3661997Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3662149Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3662522Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3662664Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3663095Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3663209Z output = model(*input) 2022-11-23T03:54:46.3663543Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3663674Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3664056Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3664222Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3664595Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3664707Z _lazy_init(state, module) 2022-11-23T03:54:46.3665066Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3665203Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3665550Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3665655Z return func(*args, **kwargs) 2022-11-23T03:54:46.3666039Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3666132Z p_assert( 2022-11-23T03:54:46.3666471Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3666585Z traceback.print_stack() 2022-11-23T03:54:46.3666803Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3666922Z File "", line 1, in 2022-11-23T03:54:46.3667120Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3667254Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3667453Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3667593Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3667794Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3667891Z self.run() 2022-11-23T03:54:46.3668086Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3668220Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3668565Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3668688Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3669048Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3669163Z getattr(self, test_name)() 2022-11-23T03:54:46.3669529Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3669673Z fn() 2022-11-23T03:54:46.3670050Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3670165Z test(self, **param_kwargs) 2022-11-23T03:54:46.3670530Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3670645Z return func(*args, **kwargs) 2022-11-23T03:54:46.3670887Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3670993Z self.run_subtests( 2022-11-23T03:54:46.3671351Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3671503Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3671930Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3672080Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3672467Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3672576Z output = model(*input) 2022-11-23T03:54:46.3672908Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3673042Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3673412Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3673578Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3673952Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3674064Z _lazy_init(state, module) 2022-11-23T03:54:46.3674427Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3674560Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3674902Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3675019Z return func(*args, **kwargs) 2022-11-23T03:54:46.3675402Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3675495Z p_assert( 2022-11-23T03:54:46.3675833Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3675948Z traceback.print_stack() 2022-11-23T03:54:46.3676168Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3676286Z File "", line 1, in 2022-11-23T03:54:46.3676488Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3676629Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3676818Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3676959Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3677151Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3677245Z self.run() 2022-11-23T03:54:46.3677437Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3677573Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3677919Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3678044Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3678414Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3678586Z getattr(self, test_name)() 2022-11-23T03:54:46.3678955Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3679043Z fn() 2022-11-23T03:54:46.3679413Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3679528Z test(self, **param_kwargs) 2022-11-23T03:54:46.3679892Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3680007Z return func(*args, **kwargs) 2022-11-23T03:54:46.3680247Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3680350Z self.run_subtests( 2022-11-23T03:54:46.3680706Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3680891Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3681270Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3681415Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3681796Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3681906Z output = model(*input) 2022-11-23T03:54:46.3682238Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3682369Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3682751Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3682915Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3683294Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3683412Z _lazy_init(state, module) 2022-11-23T03:54:46.3683769Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3683901Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3684246Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3684360Z return func(*args, **kwargs) 2022-11-23T03:54:46.3684745Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3684838Z p_assert( 2022-11-23T03:54:46.3685177Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3685293Z traceback.print_stack() 2022-11-23T03:54:46.3685498Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3685625Z File "", line 1, in 2022-11-23T03:54:46.3685823Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3685955Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3686148Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3686287Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3686489Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3686582Z self.run() 2022-11-23T03:54:46.3686775Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3686909Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3687253Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3687376Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3687916Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3688249Z getattr(self, test_name)() 2022-11-23T03:54:46.3688634Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3688723Z fn() 2022-11-23T03:54:46.3689094Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3689197Z test(self, **param_kwargs) 2022-11-23T03:54:46.3689564Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3689679Z return func(*args, **kwargs) 2022-11-23T03:54:46.3689920Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3690022Z self.run_subtests( 2022-11-23T03:54:46.3690442Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3690599Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3690973Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3691115Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3691497Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3691607Z output = model(*input) 2022-11-23T03:54:46.3691941Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3692076Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3692459Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3692624Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3693008Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3693121Z _lazy_init(state, module) 2022-11-23T03:54:46.3693479Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3693600Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3693947Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3694062Z return func(*args, **kwargs) 2022-11-23T03:54:46.3694451Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3694543Z p_assert( 2022-11-23T03:54:46.3694883Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3694998Z traceback.print_stack() 2022-11-23T03:54:46.3695232Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3695350Z File "", line 1, in 2022-11-23T03:54:46.3695550Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3695681Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3695873Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3696013Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3696217Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3696311Z self.run() 2022-11-23T03:54:46.3696504Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3696639Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3696974Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3697154Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3697527Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3697641Z getattr(self, test_name)() 2022-11-23T03:54:46.3698009Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3698096Z fn() 2022-11-23T03:54:46.3698469Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3698584Z test(self, **param_kwargs) 2022-11-23T03:54:46.3698946Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3699061Z return func(*args, **kwargs) 2022-11-23T03:54:46.3699301Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3699452Z self.run_subtests( 2022-11-23T03:54:46.3699831Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3699983Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3700354Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3700503Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3700885Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3700994Z output = model(*input) 2022-11-23T03:54:46.3701316Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3701444Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3701830Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3701997Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3702375Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3702486Z _lazy_init(state, module) 2022-11-23T03:54:46.3702842Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3702973Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3703317Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3703433Z return func(*args, **kwargs) 2022-11-23T03:54:46.3703821Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3703914Z p_assert( 2022-11-23T03:54:46.3704254Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3704372Z traceback.print_stack() 2022-11-23T03:54:46.3704593Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3704712Z File "", line 1, in 2022-11-23T03:54:46.3704911Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3705044Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3705223Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3705366Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3705569Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3705664Z self.run() 2022-11-23T03:54:46.3705855Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3705996Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3706403Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3706527Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3706894Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3707012Z getattr(self, test_name)() 2022-11-23T03:54:46.3707379Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3707470Z fn() 2022-11-23T03:54:46.3707842Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3707956Z test(self, **param_kwargs) 2022-11-23T03:54:46.3708320Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3708434Z return func(*args, **kwargs) 2022-11-23T03:54:46.3708718Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3708827Z self.run_subtests( 2022-11-23T03:54:46.3709178Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3709332Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3709701Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3709844Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3710227Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3710336Z output = model(*input) 2022-11-23T03:54:46.3710669Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3710799Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3711189Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3711354Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3711726Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3711838Z _lazy_init(state, module) 2022-11-23T03:54:46.3712193Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3712324Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3712668Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3712783Z return func(*args, **kwargs) 2022-11-23T03:54:46.3713166Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3713266Z p_assert( 2022-11-23T03:54:46.3713605Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3713709Z traceback.print_stack() 2022-11-23T03:54:46.3713929Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3714048Z File "", line 1, in 2022-11-23T03:54:46.3714247Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3714381Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3714573Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3714713Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3714917Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3715011Z self.run() 2022-11-23T03:54:46.3715201Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3715390Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3715740Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3715863Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3716232Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3716346Z getattr(self, test_name)() 2022-11-23T03:54:46.3716712Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3716801Z fn() 2022-11-23T03:54:46.3717161Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3717275Z test(self, **param_kwargs) 2022-11-23T03:54:46.3717642Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3717806Z return func(*args, **kwargs) 2022-11-23T03:54:46.3718049Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3718154Z self.run_subtests( 2022-11-23T03:54:46.3718514Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3718670Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3719039Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3719183Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3719562Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3719671Z output = model(*input) 2022-11-23T03:54:46.3720010Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3720145Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3720529Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3720695Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3721068Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3721180Z _lazy_init(state, module) 2022-11-23T03:54:46.3721538Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3721659Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3722005Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3722121Z return func(*args, **kwargs) 2022-11-23T03:54:46.3722510Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3722606Z p_assert( 2022-11-23T03:54:46.3722945Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3723060Z traceback.print_stack() 2022-11-23T03:54:46.3723281Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3723401Z File "", line 1, in 2022-11-23T03:54:46.3723597Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3723728Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3723920Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3724061Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3724263Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3724414Z self.run() 2022-11-23T03:54:46.3724608Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3724743Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3725080Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3725207Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3725578Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3725694Z getattr(self, test_name)() 2022-11-23T03:54:46.3726059Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3726149Z fn() 2022-11-23T03:54:46.3726524Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3726638Z test(self, **param_kwargs) 2022-11-23T03:54:46.3727051Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3727170Z return func(*args, **kwargs) 2022-11-23T03:54:46.3727410Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3727512Z self.run_subtests( 2022-11-23T03:54:46.3727920Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3728074Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3728447Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3728590Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3728973Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3729093Z output = model(*input) 2022-11-23T03:54:46.3729414Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3729546Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3729928Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3730093Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3730468Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3730580Z _lazy_init(state, module) 2022-11-23T03:54:46.3730938Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3731071Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3731413Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3731534Z return func(*args, **kwargs) 2022-11-23T03:54:46.3731917Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3732010Z p_assert( 2022-11-23T03:54:46.3732352Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3732466Z traceback.print_stack() 2022-11-23T03:54:46.3732684Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3732804Z File "", line 1, in 2022-11-23T03:54:46.3733002Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3733133Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3733311Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3733452Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3733720Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3733816Z self.run() 2022-11-23T03:54:46.3734007Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3734141Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3734494Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3734617Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3734986Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3735101Z getattr(self, test_name)() 2022-11-23T03:54:46.3735467Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3735556Z fn() 2022-11-23T03:54:46.3735982Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3736102Z test(self, **param_kwargs) 2022-11-23T03:54:46.3736468Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3736582Z return func(*args, **kwargs) 2022-11-23T03:54:46.3736824Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3736927Z self.run_subtests( 2022-11-23T03:54:46.3737277Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3737431Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3737806Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3737948Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3738333Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3738441Z output = model(*input) 2022-11-23T03:54:46.3738772Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3738900Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3739280Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3739440Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3739813Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3739923Z _lazy_init(state, module) 2022-11-23T03:54:46.3740285Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3740417Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3740767Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3740886Z return func(*args, **kwargs) 2022-11-23T03:54:46.3741271Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3741364Z p_assert( 2022-11-23T03:54:46.3741705Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3741809Z traceback.print_stack() 2022-11-23T03:54:46.3742028Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3742146Z File "", line 1, in 2022-11-23T03:54:46.3742343Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3742473Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3742667Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3742885Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3743090Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3743188Z self.run() 2022-11-23T03:54:46.3743378Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3743513Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3743863Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3743987Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3744355Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3744471Z getattr(self, test_name)() 2022-11-23T03:54:46.3744837Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3744932Z fn() 2022-11-23T03:54:46.3745337Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3745457Z test(self, **param_kwargs) 2022-11-23T03:54:46.3745828Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3745944Z return func(*args, **kwargs) 2022-11-23T03:54:46.3746183Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3746286Z self.run_subtests( 2022-11-23T03:54:46.3746643Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3746795Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3747164Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3747314Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3747696Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3747805Z output = model(*input) 2022-11-23T03:54:46.3748139Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3748271Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3748652Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3748816Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3749190Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3749304Z _lazy_init(state, module) 2022-11-23T03:54:46.3749663Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3749787Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3750129Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3750245Z return func(*args, **kwargs) 2022-11-23T03:54:46.3750628Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3750723Z p_assert( 2022-11-23T03:54:46.3751062Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3751176Z traceback.print_stack() 2022-11-23T03:54:46.3751394Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3751512Z File "", line 1, in 2022-11-23T03:54:46.3751707Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3751895Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3752087Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3752228Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3752429Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3752524Z self.run() 2022-11-23T03:54:46.3752718Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3752851Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3753188Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3753316Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3753684Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3753798Z getattr(self, test_name)() 2022-11-23T03:54:46.3754210Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3754307Z fn() 2022-11-23T03:54:46.3754683Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3754799Z test(self, **param_kwargs) 2022-11-23T03:54:46.3755163Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3755283Z return func(*args, **kwargs) 2022-11-23T03:54:46.3755526Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3755631Z self.run_subtests( 2022-11-23T03:54:46.3755988Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3756138Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3756511Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3756655Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3757036Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3757145Z output = model(*input) 2022-11-23T03:54:46.3757466Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3757599Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3757981Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3758146Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3758520Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3758633Z _lazy_init(state, module) 2022-11-23T03:54:46.3758998Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3759132Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3759476Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3759595Z return func(*args, **kwargs) 2022-11-23T03:54:46.3759980Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3760071Z p_assert( 2022-11-23T03:54:46.3760410Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3760525Z traceback.print_stack() 2022-11-23T03:54:46.3760745Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3760864Z File "", line 1, in 2022-11-23T03:54:46.3761122Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3761255Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3761435Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3761577Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3761779Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3761874Z self.run() 2022-11-23T03:54:46.3762067Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3762202Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3762552Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3762675Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3763043Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3763213Z getattr(self, test_name)() 2022-11-23T03:54:46.3763586Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3763675Z fn() 2022-11-23T03:54:46.3764049Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3764165Z test(self, **param_kwargs) 2022-11-23T03:54:46.3764529Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3764644Z return func(*args, **kwargs) 2022-11-23T03:54:46.3764883Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3764975Z self.run_subtests( 2022-11-23T03:54:46.3765334Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3765493Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3765863Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3766006Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3766388Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3766494Z output = model(*input) 2022-11-23T03:54:46.3766828Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3766959Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3767342Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3767508Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3768008Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3768268Z _lazy_init(state, module) 2022-11-23T03:54:46.3768644Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3768778Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3769120Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3769235Z return func(*args, **kwargs) 2022-11-23T03:54:46.3769618Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3769709Z p_assert( 2022-11-23T03:54:46.3770034Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3770151Z traceback.print_stack() 2022-11-23T03:54:46.3770371Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3770570Z File "", line 1, in 2022-11-23T03:54:46.3770767Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3770899Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3771092Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3771235Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3771434Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3771530Z self.run() 2022-11-23T03:54:46.3771723Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3771856Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3772206Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3772329Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3772753Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3772870Z getattr(self, test_name)() 2022-11-23T03:54:46.3773238Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3773316Z fn() 2022-11-23T03:54:46.3773688Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3773803Z test(self, **param_kwargs) 2022-11-23T03:54:46.3774169Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3774285Z return func(*args, **kwargs) 2022-11-23T03:54:46.3774528Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3774635Z self.run_subtests( 2022-11-23T03:54:46.3774997Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3775156Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3775525Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3775670Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3776051Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3776160Z output = model(*input) 2022-11-23T03:54:46.3776491Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3776623Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3777005Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3777172Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3777547Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3777660Z _lazy_init(state, module) 2022-11-23T03:54:46.3778002Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3778134Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3778481Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3778598Z return func(*args, **kwargs) 2022-11-23T03:54:46.3778982Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3779074Z p_assert( 2022-11-23T03:54:46.3779413Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3779592Z traceback.print_stack() 2022-11-23T03:54:46.3779811Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3779930Z File "", line 1, in 2022-11-23T03:54:46.3780141Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3780276Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3780466Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3780607Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3780811Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3780907Z self.run() 2022-11-23T03:54:46.3781099Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3781223Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3781621Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3781754Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3782125Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3782241Z getattr(self, test_name)() 2022-11-23T03:54:46.3782608Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3782697Z fn() 2022-11-23T03:54:46.3783066Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3783182Z test(self, **param_kwargs) 2022-11-23T03:54:46.3783544Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3783659Z return func(*args, **kwargs) 2022-11-23T03:54:46.3783900Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3784009Z self.run_subtests( 2022-11-23T03:54:46.3784365Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3784518Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3784887Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3785030Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3785413Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3785510Z output = model(*input) 2022-11-23T03:54:46.3785843Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3785975Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3786360Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3786527Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3786899Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3787014Z _lazy_init(state, module) 2022-11-23T03:54:46.3787369Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3787501Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3787844Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3787960Z return func(*args, **kwargs) 2022-11-23T03:54:46.3788343Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3788436Z p_assert( 2022-11-23T03:54:46.3788777Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3788949Z traceback.print_stack() 2022-11-23T03:54:46.3789171Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3789290Z File "", line 1, in 2022-11-23T03:54:46.3789488Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3789609Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3789803Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3789946Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3790149Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3790245Z self.run() 2022-11-23T03:54:46.3790437Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3790570Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3790965Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3791091Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3791462Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3791577Z getattr(self, test_name)() 2022-11-23T03:54:46.3791942Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3792031Z fn() 2022-11-23T03:54:46.3792403Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3792522Z test(self, **param_kwargs) 2022-11-23T03:54:46.3792885Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3793001Z return func(*args, **kwargs) 2022-11-23T03:54:46.3793236Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 150, in test_nested_always_wrap_model 2022-11-23T03:54:46.3793341Z self.run_subtests( 2022-11-23T03:54:46.3793698Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3793850Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3794218Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3794362Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3794744Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3794854Z output = model(*input) 2022-11-23T03:54:46.3795185Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3795321Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3795704Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3795871Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3796244Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3796357Z _lazy_init(state, module) 2022-11-23T03:54:46.3796713Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 93, in _lazy_init 2022-11-23T03:54:46.3796845Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3797186Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3797301Z return func(*args, **kwargs) 2022-11-23T03:54:46.3797684Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3797818Z p_assert( 2022-11-23T03:54:46.3798161Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3798276Z traceback.print_stack() 2022-11-23T03:54:46.3798494Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3798709Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3798925Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3799143Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3799359Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3799573Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3799831Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3800057Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3800272Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3800484Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3800700Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3800914Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3801129Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3801342Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3801554Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3801772Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3801985Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3802186Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3802404Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3802620Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3802838Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3803050Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3803263Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3803476Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3803694Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3803907Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3804123Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3804335Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3804547Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3804761Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3804972Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3805188Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3805403Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3805661Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3805873Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3806085Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3806297Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3806511Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3806711Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3806928Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3807143Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3807411Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3807633Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3807968Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3808184Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3808398Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3808612Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3808713Z dist init r=0, world=2 2022-11-23T03:54:46.3809028Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.3809345Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.3809654Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.3809959Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.3810260Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.3810559Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.3810859Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.3811164Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.3811464Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.3811760Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.3812058Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.3812355Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.3812460Z dist init r=1, world=2 2022-11-23T03:54:46.3812831Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.3813134Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.3813434Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.3813734Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.3814032Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.3814374Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.3814681Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.3814982Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.3815284Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.3815582Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.3815884Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.3816182Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.3816277Z ok (9.340s) 2022-11-23T03:54:46.3816597Z test_nested_wrapped_model_offload_false_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49176 2022-11-23T03:54:46.3816810Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49177 2022-11-23T03:54:46.3817192Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.3817359Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.3817747Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.3817928Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.3818157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.3818535Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.3818699Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.3819088Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.3819265Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.3819490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.3819886Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.3820285Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.3820620Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.3820902Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.3821117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.3821332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.3821552Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3821771Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3822902Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.3823012Z warnings.warn( 2022-11-23T03:54:46.3823234Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3824283Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.3824384Z warnings.warn( 2022-11-23T03:54:46.3824602Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3824821Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3825036Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3825253Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3825468Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3825670Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3825884Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3826098Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3826312Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3826528Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3826744Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3826960Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3827172Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3827385Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3827598Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3827809Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3828026Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3828239Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3828472Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3828738Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3828951Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3829166Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3829377Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3829590Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3829803Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3830016Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3830217Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3830473Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3830689Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3830906Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3831117Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3831333Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3831545Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3831759Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3831970Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3832186Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3832408Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3832622Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3832834Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3833047Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3833260Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3833473Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3833685Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3834478Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.3835253Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.3836015Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.3836788Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.3837613Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.3838380Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.3839180Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.3839951Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.3840170Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3840389Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3840491Z dist init r=0, world=2 2022-11-23T03:54:46.3840598Z dist init r=1, world=2 2022-11-23T03:54:46.3840692Z ok (8.440s) 2022-11-23T03:54:46.3841008Z test_nested_wrapped_model_offload_false_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49329 2022-11-23T03:54:46.3841223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49330 2022-11-23T03:54:46.3841601Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.3841770Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.3842156Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.3842334Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.3842557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.3842931Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.3843096Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.3843471Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.3843652Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.3843882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.3844281Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.3844675Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.3844949Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.3845225Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.3845438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.3845656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.3845929Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3846151Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3847207Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.3847309Z warnings.warn( 2022-11-23T03:54:46.3847528Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3849028Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.3849139Z warnings.warn( 2022-11-23T03:54:46.3849361Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3849578Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3849794Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3850010Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3850226Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3850448Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3850660Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3850873Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3851086Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3851302Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3851515Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3851732Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3851933Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3852150Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3852366Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3852580Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3852792Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3853005Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3853216Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3853433Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3853647Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3853860Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3854073Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3854337Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3854548Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3854761Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3854972Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3855188Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3855399Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3855612Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3855824Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3856078Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3856293Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3856494Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3856707Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3856919Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3857135Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3857347Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3857558Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3857772Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3857989Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3858201Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3858417Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3858630Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3858844Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3858948Z dist init r=0, world=2 2022-11-23T03:54:46.3859050Z dist init r=1, world=2 2022-11-23T03:54:46.3859142Z ok (8.740s) 2022-11-23T03:54:46.3859467Z test_nested_wrapped_model_offload_false_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49482 2022-11-23T03:54:46.3859678Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49483 2022-11-23T03:54:46.3860073Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.3860237Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.3860621Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.3860788Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.3861013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.3861386Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.3861551Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.3861935Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.3862170Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.3862395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.3862795Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.3863189Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.3863466Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.3863743Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.3863957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.3864170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.3864431Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3864653Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3865713Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.3865816Z warnings.warn( 2022-11-23T03:54:46.3866031Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3867087Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.3867189Z warnings.warn( 2022-11-23T03:54:46.3867406Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3867625Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3867838Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3868053Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3868264Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3868481Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3868696Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3868910Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3869111Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3869325Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3869536Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3869750Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3869963Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3870176Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3870392Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3870671Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3870887Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3871099Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3871313Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3871526Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3871739Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3871953Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3872166Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3872419Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3872636Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3872849Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3873065Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3873277Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3873490Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3873691Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3873905Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3874118Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3874340Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3874555Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3874772Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3874986Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3875199Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3875411Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3875625Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3875839Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3876054Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3876268Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3876480Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3876694Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3876906Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3877009Z dist init r=0, world=2 2022-11-23T03:54:46.3877111Z dist init r=1, world=2 2022-11-23T03:54:46.3877203Z ok (8.635s) 2022-11-23T03:54:46.3877505Z test_nested_wrapped_model_offload_true_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49635 2022-11-23T03:54:46.3877713Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49636 2022-11-23T03:54:46.3878100Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.3878317Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.3878705Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.3878885Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.3879107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.3879480Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.3879644Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.3880025Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.3880201Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.3880475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.3880879Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.3881271Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.3881547Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.3881820Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.3882034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.3882249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.3882466Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3882686Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3883726Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.3883827Z warnings.warn( 2022-11-23T03:54:46.3883948Z File "", line 1, in 2022-11-23T03:54:46.3884146Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3884279Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3884458Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3884605Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3884810Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3884909Z self.run() 2022-11-23T03:54:46.3885101Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3885234Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3885581Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3885705Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3886081Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3886197Z getattr(self, test_name)() 2022-11-23T03:54:46.3886566Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3886655Z fn() 2022-11-23T03:54:46.3887030Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3887200Z test(self, **param_kwargs) 2022-11-23T03:54:46.3887565Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3887680Z return func(*args, **kwargs) 2022-11-23T03:54:46.3887972Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.3888075Z self.run_subtests( 2022-11-23T03:54:46.3888424Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3888576Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3888949Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3889092Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3889532Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3889646Z output = model(*input) 2022-11-23T03:54:46.3889984Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3890115Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3890498Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3890661Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3891037Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3891149Z _lazy_init(state, module) 2022-11-23T03:54:46.3891504Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.3891642Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3891988Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3892103Z return func(*args, **kwargs) 2022-11-23T03:54:46.3892487Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3892580Z p_assert( 2022-11-23T03:54:46.3892919Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3893022Z traceback.print_stack() 2022-11-23T03:54:46.3893239Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3894296Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.3894413Z warnings.warn( 2022-11-23T03:54:46.3894521Z File "", line 1, in 2022-11-23T03:54:46.3894720Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3894853Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3895043Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3895183Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3895387Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3895481Z self.run() 2022-11-23T03:54:46.3895677Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3895811Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3896229Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3896352Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3896721Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3896835Z getattr(self, test_name)() 2022-11-23T03:54:46.3897200Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3897289Z fn() 2022-11-23T03:54:46.3897663Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3897767Z test(self, **param_kwargs) 2022-11-23T03:54:46.3898132Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3898248Z return func(*args, **kwargs) 2022-11-23T03:54:46.3898537Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.3898643Z self.run_subtests( 2022-11-23T03:54:46.3899005Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3899156Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3899530Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3899673Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3900057Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3900167Z output = model(*input) 2022-11-23T03:54:46.3900500Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3900636Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3901023Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3901187Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3901560Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3901672Z _lazy_init(state, module) 2022-11-23T03:54:46.3902030Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.3902161Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3902493Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3902612Z return func(*args, **kwargs) 2022-11-23T03:54:46.3902997Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3903096Z p_assert( 2022-11-23T03:54:46.3903436Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3903551Z traceback.print_stack() 2022-11-23T03:54:46.3903768Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3903889Z File "", line 1, in 2022-11-23T03:54:46.3904088Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3904221Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3904410Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3904549Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3904751Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3904847Z self.run() 2022-11-23T03:54:46.3905047Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3905238Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3905601Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3905713Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3906085Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3906199Z getattr(self, test_name)() 2022-11-23T03:54:46.3906567Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3906656Z fn() 2022-11-23T03:54:46.3907033Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3907149Z test(self, **param_kwargs) 2022-11-23T03:54:46.3907559Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3907682Z return func(*args, **kwargs) 2022-11-23T03:54:46.3907919Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.3908024Z self.run_subtests( 2022-11-23T03:54:46.3908385Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3908536Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3908909Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3909051Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3909435Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3909548Z output = model(*input) 2022-11-23T03:54:46.3909885Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3910008Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3910393Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3910560Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3910936Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3911050Z _lazy_init(state, module) 2022-11-23T03:54:46.3911414Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.3911550Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3911894Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3912010Z return func(*args, **kwargs) 2022-11-23T03:54:46.3912401Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3912495Z p_assert( 2022-11-23T03:54:46.3912835Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3912951Z traceback.print_stack() 2022-11-23T03:54:46.3913169Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3913288Z File "", line 1, in 2022-11-23T03:54:46.3913488Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3913621Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3913811Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3913939Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3914143Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3914301Z self.run() 2022-11-23T03:54:46.3914496Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3914632Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3914981Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3915106Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3915472Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3915592Z getattr(self, test_name)() 2022-11-23T03:54:46.3915957Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3916049Z fn() 2022-11-23T03:54:46.3916422Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3916536Z test(self, **param_kwargs) 2022-11-23T03:54:46.3916954Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3917073Z return func(*args, **kwargs) 2022-11-23T03:54:46.3917311Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.3917414Z self.run_subtests( 2022-11-23T03:54:46.3917762Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3917915Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3918284Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3918428Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3918810Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3918928Z output = model(*input) 2022-11-23T03:54:46.3919259Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3919390Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3919776Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3919943Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3920316Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3920432Z _lazy_init(state, module) 2022-11-23T03:54:46.3920790Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.3920921Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3921269Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3921388Z return func(*args, **kwargs) 2022-11-23T03:54:46.3921775Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3921869Z p_assert( 2022-11-23T03:54:46.3922208Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3922311Z traceback.print_stack() 2022-11-23T03:54:46.3922528Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3922648Z File "", line 1, in 2022-11-23T03:54:46.3922848Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3922981Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3923172Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3923314Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3923581Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3923676Z self.run() 2022-11-23T03:54:46.3923868Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3924005Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3924356Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3924485Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3924854Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3924968Z getattr(self, test_name)() 2022-11-23T03:54:46.3925338Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3925428Z fn() 2022-11-23T03:54:46.3925835Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3925959Z test(self, **param_kwargs) 2022-11-23T03:54:46.3926325Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3926441Z return func(*args, **kwargs) 2022-11-23T03:54:46.3926679Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.3926783Z self.run_subtests( 2022-11-23T03:54:46.3927143Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3927295Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3927666Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3928018Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3928530Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3928644Z output = model(*input) 2022-11-23T03:54:46.3928980Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3929113Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3929496Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3929660Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3930034Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3930146Z _lazy_init(state, module) 2022-11-23T03:54:46.3930506Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.3930627Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3930976Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3931093Z return func(*args, **kwargs) 2022-11-23T03:54:46.3931478Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3931572Z p_assert( 2022-11-23T03:54:46.3931913Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3932029Z traceback.print_stack() 2022-11-23T03:54:46.3932250Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3932380Z File "", line 1, in 2022-11-23T03:54:46.3932580Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3932712Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3932907Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3933131Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3933336Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3933431Z self.run() 2022-11-23T03:54:46.3933622Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3933760Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3934099Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3934222Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3934592Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3934707Z getattr(self, test_name)() 2022-11-23T03:54:46.3935085Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3935275Z fn() 2022-11-23T03:54:46.3935654Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3935770Z test(self, **param_kwargs) 2022-11-23T03:54:46.3936133Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3936248Z return func(*args, **kwargs) 2022-11-23T03:54:46.3936484Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.3936591Z self.run_subtests( 2022-11-23T03:54:46.3936958Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3937109Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3937479Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3937628Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3938010Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3938120Z output = model(*input) 2022-11-23T03:54:46.3938441Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3938577Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3938960Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3939131Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3939505Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3939617Z _lazy_init(state, module) 2022-11-23T03:54:46.3939978Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.3940113Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3940456Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3940574Z return func(*args, **kwargs) 2022-11-23T03:54:46.3940959Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3941053Z p_assert( 2022-11-23T03:54:46.3941392Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3941509Z traceback.print_stack() 2022-11-23T03:54:46.3941727Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3941846Z File "", line 1, in 2022-11-23T03:54:46.3942046Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3942242Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3942419Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3942561Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3942767Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3942866Z self.run() 2022-11-23T03:54:46.3943057Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3943193Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3943548Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3943674Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3944044Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3944158Z getattr(self, test_name)() 2022-11-23T03:54:46.3944568Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3944663Z fn() 2022-11-23T03:54:46.3945036Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3945151Z test(self, **param_kwargs) 2022-11-23T03:54:46.3945524Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3945639Z return func(*args, **kwargs) 2022-11-23T03:54:46.3945875Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.3945967Z self.run_subtests( 2022-11-23T03:54:46.3946325Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3946478Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3946855Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3947002Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3947384Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3947495Z output = model(*input) 2022-11-23T03:54:46.3947828Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3947960Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3948351Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3948518Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3948891Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3949013Z _lazy_init(state, module) 2022-11-23T03:54:46.3949370Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.3949503Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3949847Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3949963Z return func(*args, **kwargs) 2022-11-23T03:54:46.3950356Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3950450Z p_assert( 2022-11-23T03:54:46.3950779Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3950894Z traceback.print_stack() 2022-11-23T03:54:46.3951112Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3951236Z File "", line 1, in 2022-11-23T03:54:46.3951495Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3951628Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3951820Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3951962Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3952165Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3952264Z self.run() 2022-11-23T03:54:46.3952457Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3952591Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3952943Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3953067Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3953437Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3953603Z getattr(self, test_name)() 2022-11-23T03:54:46.3953961Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3954051Z fn() 2022-11-23T03:54:46.3954422Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3954538Z test(self, **param_kwargs) 2022-11-23T03:54:46.3954901Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3955019Z return func(*args, **kwargs) 2022-11-23T03:54:46.3955257Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.3955361Z self.run_subtests( 2022-11-23T03:54:46.3955717Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3955876Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3956248Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3956391Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3956778Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3956889Z output = model(*input) 2022-11-23T03:54:46.3957224Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3957362Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3957749Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3957914Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3958293Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3958397Z _lazy_init(state, module) 2022-11-23T03:54:46.3958756Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.3958890Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3959235Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3959355Z return func(*args, **kwargs) 2022-11-23T03:54:46.3959740Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3959834Z p_assert( 2022-11-23T03:54:46.3960175Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3960293Z traceback.print_stack() 2022-11-23T03:54:46.3960518Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3960698Z File "", line 1, in 2022-11-23T03:54:46.3960898Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3961035Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3961228Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3961369Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3961575Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3961657Z self.run() 2022-11-23T03:54:46.3961854Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3961990Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3962339Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3962464Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3962881Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3963001Z getattr(self, test_name)() 2022-11-23T03:54:46.3963371Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3963461Z fn() 2022-11-23T03:54:46.3963834Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3963950Z test(self, **param_kwargs) 2022-11-23T03:54:46.3964315Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3964431Z return func(*args, **kwargs) 2022-11-23T03:54:46.3964667Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.3964771Z self.run_subtests( 2022-11-23T03:54:46.3965131Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3965287Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3965658Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3965791Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3966174Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3966286Z output = model(*input) 2022-11-23T03:54:46.3966619Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3966753Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3967135Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3967302Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3967681Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3967855Z _lazy_init(state, module) 2022-11-23T03:54:46.3968215Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.3968348Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3968699Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3968815Z return func(*args, **kwargs) 2022-11-23T03:54:46.3969198Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3969300Z p_assert( 2022-11-23T03:54:46.3969641Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3969827Z traceback.print_stack() 2022-11-23T03:54:46.3970054Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3970166Z File "", line 1, in 2022-11-23T03:54:46.3970413Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3970575Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3970813Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3970994Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3971240Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3971351Z self.run() 2022-11-23T03:54:46.3971596Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3971760Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3972252Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3972414Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3972862Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3973004Z getattr(self, test_name)() 2022-11-23T03:54:46.3973444Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3973549Z fn() 2022-11-23T03:54:46.3974001Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3974137Z test(self, **param_kwargs) 2022-11-23T03:54:46.3974565Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3974703Z return func(*args, **kwargs) 2022-11-23T03:54:46.3974992Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.3975121Z self.run_subtests( 2022-11-23T03:54:46.3975560Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3975741Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3976189Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3976366Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3976830Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3976959Z output = model(*input) 2022-11-23T03:54:46.3977361Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3977529Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3977996Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3978201Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3978651Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3978786Z _lazy_init(state, module) 2022-11-23T03:54:46.3979216Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.3979375Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3979787Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3979916Z return func(*args, **kwargs) 2022-11-23T03:54:46.3980383Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3980493Z p_assert( 2022-11-23T03:54:46.3980911Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3981124Z traceback.print_stack() 2022-11-23T03:54:46.3981388Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3981530Z File "", line 1, in 2022-11-23T03:54:46.3981768Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3981928Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3982160Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3982333Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3982581Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3982692Z self.run() 2022-11-23T03:54:46.3982925Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3983086Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3983558Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3983713Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3984152Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3984287Z getattr(self, test_name)() 2022-11-23T03:54:46.3984730Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3984837Z fn() 2022-11-23T03:54:46.3985287Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3985426Z test(self, **param_kwargs) 2022-11-23T03:54:46.3985865Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3986004Z return func(*args, **kwargs) 2022-11-23T03:54:46.3986294Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.3986416Z self.run_subtests( 2022-11-23T03:54:46.3986845Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3987029Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3987477Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3987650Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3988112Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3988248Z output = model(*input) 2022-11-23T03:54:46.3988582Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3988718Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3989093Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3989264Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3989645Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3989759Z _lazy_init(state, module) 2022-11-23T03:54:46.3990118Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.3990249Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.3990594Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.3990709Z return func(*args, **kwargs) 2022-11-23T03:54:46.3991094Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.3991251Z p_assert( 2022-11-23T03:54:46.3991598Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.3991715Z traceback.print_stack() 2022-11-23T03:54:46.3991936Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.3992057Z File "", line 1, in 2022-11-23T03:54:46.3992258Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.3992392Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.3992584Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.3992713Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.3992922Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.3993018Z self.run() 2022-11-23T03:54:46.3993267Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.3993413Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.3993764Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.3993889Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.3994258Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.3994376Z getattr(self, test_name)() 2022-11-23T03:54:46.3994744Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.3994836Z fn() 2022-11-23T03:54:46.3995208Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.3995323Z test(self, **param_kwargs) 2022-11-23T03:54:46.3995691Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.3995811Z return func(*args, **kwargs) 2022-11-23T03:54:46.3996047Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.3996151Z self.run_subtests( 2022-11-23T03:54:46.3996497Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.3996650Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.3997033Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.3997176Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.3997559Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.3997672Z output = model(*input) 2022-11-23T03:54:46.3998009Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.3998147Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.3998533Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.3998705Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.3999081Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.3999196Z _lazy_init(state, module) 2022-11-23T03:54:46.3999560Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.3999694Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4000041Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4000162Z return func(*args, **kwargs) 2022-11-23T03:54:46.4000642Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4000724Z p_assert( 2022-11-23T03:54:46.4001066Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4001186Z traceback.print_stack() 2022-11-23T03:54:46.4001409Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4001530Z File "", line 1, in 2022-11-23T03:54:46.4001727Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4001862Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4002057Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4002201Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4002406Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4002552Z self.run() 2022-11-23T03:54:46.4002757Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4002893Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4003243Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4003373Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4003744Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4003862Z getattr(self, test_name)() 2022-11-23T03:54:46.4004215Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4004309Z fn() 2022-11-23T03:54:46.4004682Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4004799Z test(self, **param_kwargs) 2022-11-23T03:54:46.4005174Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4005292Z return func(*args, **kwargs) 2022-11-23T03:54:46.4005528Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4005632Z self.run_subtests( 2022-11-23T03:54:46.4005991Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4006149Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4006521Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4006669Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4007054Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4007170Z output = model(*input) 2022-11-23T03:54:46.4007513Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4007646Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4008311Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4008481Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4008847Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4008962Z _lazy_init(state, module) 2022-11-23T03:54:46.4009321Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4009454Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4009805Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4010016Z return func(*args, **kwargs) 2022-11-23T03:54:46.4010403Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4010498Z p_assert( 2022-11-23T03:54:46.4010840Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4010957Z traceback.print_stack() 2022-11-23T03:54:46.4011177Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4011297Z File "", line 1, in 2022-11-23T03:54:46.4011501Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4011634Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4011827Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4011968Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4012227Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4012313Z self.run() 2022-11-23T03:54:46.4012511Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4012648Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4013002Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4013127Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4013499Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4013616Z getattr(self, test_name)() 2022-11-23T03:54:46.4013982Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4014073Z fn() 2022-11-23T03:54:46.4014449Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4014571Z test(self, **param_kwargs) 2022-11-23T03:54:46.4014939Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4015056Z return func(*args, **kwargs) 2022-11-23T03:54:46.4015294Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4015401Z self.run_subtests( 2022-11-23T03:54:46.4015757Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4015914Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4016286Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4016417Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4016805Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4016920Z output = model(*input) 2022-11-23T03:54:46.4017253Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4017387Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4017773Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4017940Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4018316Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4018430Z _lazy_init(state, module) 2022-11-23T03:54:46.4018788Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4018922Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4019337Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4019455Z return func(*args, **kwargs) 2022-11-23T03:54:46.4019840Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4019938Z p_assert( 2022-11-23T03:54:46.4020278Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4020399Z traceback.print_stack() 2022-11-23T03:54:46.4020622Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4020729Z File "", line 1, in 2022-11-23T03:54:46.4020930Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4021065Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4021300Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4021451Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4021656Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4021751Z self.run() 2022-11-23T03:54:46.4021945Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4022080Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4022430Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4022558Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4022929Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4023045Z getattr(self, test_name)() 2022-11-23T03:54:46.4023412Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4023510Z fn() 2022-11-23T03:54:46.4023883Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4023987Z test(self, **param_kwargs) 2022-11-23T03:54:46.4024351Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4024471Z return func(*args, **kwargs) 2022-11-23T03:54:46.4024709Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4024819Z self.run_subtests( 2022-11-23T03:54:46.4025178Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4025332Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4025702Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4025852Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4026236Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4026349Z output = model(*input) 2022-11-23T03:54:46.4026683Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4026819Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4027203Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4027368Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4027750Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4027865Z _lazy_init(state, module) 2022-11-23T03:54:46.4028229Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4028424Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4028758Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4028877Z return func(*args, **kwargs) 2022-11-23T03:54:46.4029260Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4029359Z p_assert( 2022-11-23T03:54:46.4029701Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4029819Z traceback.print_stack() 2022-11-23T03:54:46.4030038Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4030161Z File "", line 1, in 2022-11-23T03:54:46.4030361Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4030551Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4030748Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4030892Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4031098Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4031196Z self.run() 2022-11-23T03:54:46.4031391Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4031526Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4031878Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4031991Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4032361Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4032479Z getattr(self, test_name)() 2022-11-23T03:54:46.4032850Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4032946Z fn() 2022-11-23T03:54:46.4033319Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4033436Z test(self, **param_kwargs) 2022-11-23T03:54:46.4033807Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4033925Z return func(*args, **kwargs) 2022-11-23T03:54:46.4034163Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4034270Z self.run_subtests( 2022-11-23T03:54:46.4034633Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4034789Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4035165Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4035313Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4035698Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4035810Z output = model(*input) 2022-11-23T03:54:46.4036147Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4036268Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4036651Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4036820Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4037195Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4037367Z _lazy_init(state, module) 2022-11-23T03:54:46.4037733Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4037871Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4038216Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4038333Z return func(*args, **kwargs) 2022-11-23T03:54:46.4038722Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4038817Z p_assert( 2022-11-23T03:54:46.4039158Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4039277Z traceback.print_stack() 2022-11-23T03:54:46.4039501Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4039624Z File "", line 1, in 2022-11-23T03:54:46.4039875Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4040013Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4040193Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4040337Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4040543Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4040639Z self.run() 2022-11-23T03:54:46.4040833Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4040969Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4041319Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4041448Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4041817Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4041942Z getattr(self, test_name)() 2022-11-23T03:54:46.4042312Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4042403Z fn() 2022-11-23T03:54:46.4042775Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4042893Z test(self, **param_kwargs) 2022-11-23T03:54:46.4043255Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4043370Z return func(*args, **kwargs) 2022-11-23T03:54:46.4043611Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4043702Z self.run_subtests( 2022-11-23T03:54:46.4044063Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4044220Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4044592Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4044738Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4045124Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4045235Z output = model(*input) 2022-11-23T03:54:46.4045571Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4045707Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4046092Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4046260Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4046640Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4046820Z _lazy_init(state, module) 2022-11-23T03:54:46.4047185Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4047318Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4047663Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4047849Z return func(*args, **kwargs) 2022-11-23T03:54:46.4048235Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4048331Z p_assert( 2022-11-23T03:54:46.4048659Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4048776Z traceback.print_stack() 2022-11-23T03:54:46.4049053Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4049179Z File "", line 1, in 2022-11-23T03:54:46.4049383Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4049515Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4049707Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4049850Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4050055Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4050153Z self.run() 2022-11-23T03:54:46.4050345Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4050484Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4050835Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4050962Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4051346Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4051462Z getattr(self, test_name)() 2022-11-23T03:54:46.4051828Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4051905Z fn() 2022-11-23T03:54:46.4052276Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4052390Z test(self, **param_kwargs) 2022-11-23T03:54:46.4052755Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4052872Z return func(*args, **kwargs) 2022-11-23T03:54:46.4053112Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4053218Z self.run_subtests( 2022-11-23T03:54:46.4053581Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4053737Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4054112Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4054256Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4054640Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4054758Z output = model(*input) 2022-11-23T03:54:46.4055094Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4055229Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4055612Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4055784Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4056227Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4056327Z _lazy_init(state, module) 2022-11-23T03:54:46.4056689Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4056827Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4057172Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4057290Z return func(*args, **kwargs) 2022-11-23T03:54:46.4057679Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4057776Z p_assert( 2022-11-23T03:54:46.4058118Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4058292Z traceback.print_stack() 2022-11-23T03:54:46.4058516Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4058639Z File "", line 1, in 2022-11-23T03:54:46.4058838Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4058974Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4059169Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4059312Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4059515Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4059613Z self.run() 2022-11-23T03:54:46.4059793Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4059931Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4060286Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4060414Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4060787Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4060909Z getattr(self, test_name)() 2022-11-23T03:54:46.4061277Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4061369Z fn() 2022-11-23T03:54:46.4061741Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4061857Z test(self, **param_kwargs) 2022-11-23T03:54:46.4062220Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4062339Z return func(*args, **kwargs) 2022-11-23T03:54:46.4062578Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4062690Z self.run_subtests( 2022-11-23T03:54:46.4063048Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4063200Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4063574Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4063720Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4064090Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4064203Z output = model(*input) 2022-11-23T03:54:46.4064542Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4064678Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4065065Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4065291Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4065671Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4065788Z _lazy_init(state, module) 2022-11-23T03:54:46.4066149Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4066285Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4066631Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4066749Z return func(*args, **kwargs) 2022-11-23T03:54:46.4067140Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4067237Z p_assert( 2022-11-23T03:54:46.4067628Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4067752Z traceback.print_stack() 2022-11-23T03:54:46.4067974Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4068095Z File "", line 1, in 2022-11-23T03:54:46.4068280Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4068415Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4068611Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4068755Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4068958Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4069055Z self.run() 2022-11-23T03:54:46.4069250Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4069386Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4069743Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4069874Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4070245Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4070365Z getattr(self, test_name)() 2022-11-23T03:54:46.4070730Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4070821Z fn() 2022-11-23T03:54:46.4071195Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4071312Z test(self, **param_kwargs) 2022-11-23T03:54:46.4071676Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4071779Z return func(*args, **kwargs) 2022-11-23T03:54:46.4072023Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4072129Z self.run_subtests( 2022-11-23T03:54:46.4072486Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4072644Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4073013Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4073159Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4073543Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4073655Z output = model(*input) 2022-11-23T03:54:46.4073992Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4074129Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4074574Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4074741Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4075117Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4075232Z _lazy_init(state, module) 2022-11-23T03:54:46.4075592Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4075726Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4076070Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4076173Z return func(*args, **kwargs) 2022-11-23T03:54:46.4076619Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4076723Z p_assert( 2022-11-23T03:54:46.4077066Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4077184Z traceback.print_stack() 2022-11-23T03:54:46.4077407Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4077528Z File "", line 1, in 2022-11-23T03:54:46.4077728Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4077864Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4078054Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4078198Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4078402Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4078498Z self.run() 2022-11-23T03:54:46.4078694Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4078839Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4079186Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4079312Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4079668Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4079788Z getattr(self, test_name)() 2022-11-23T03:54:46.4080155Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4080249Z fn() 2022-11-23T03:54:46.4080623Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4080744Z test(self, **param_kwargs) 2022-11-23T03:54:46.4081112Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4081232Z return func(*args, **kwargs) 2022-11-23T03:54:46.4081471Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4081577Z self.run_subtests( 2022-11-23T03:54:46.4081938Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4082093Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4082465Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4082609Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4082994Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4083106Z output = model(*input) 2022-11-23T03:54:46.4083446Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4083636Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4084010Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4084179Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4084552Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4084667Z _lazy_init(state, module) 2022-11-23T03:54:46.4085025Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4085161Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4085508Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4085626Z return func(*args, **kwargs) 2022-11-23T03:54:46.4086064Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4086164Z p_assert( 2022-11-23T03:54:46.4086506Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4086624Z traceback.print_stack() 2022-11-23T03:54:46.4086842Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4086966Z File "", line 1, in 2022-11-23T03:54:46.4087164Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4087297Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4087490Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4087619Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4087932Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4088104Z self.run() 2022-11-23T03:54:46.4088402Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4088541Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4088901Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4089029Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4089400Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4089516Z getattr(self, test_name)() 2022-11-23T03:54:46.4089883Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4089973Z fn() 2022-11-23T03:54:46.4090345Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4090461Z test(self, **param_kwargs) 2022-11-23T03:54:46.4090831Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4090950Z return func(*args, **kwargs) 2022-11-23T03:54:46.4091191Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4091296Z self.run_subtests( 2022-11-23T03:54:46.4091642Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4091797Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4092169Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4092313Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4092699Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4092895Z output = model(*input) 2022-11-23T03:54:46.4093237Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4093370Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4093753Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4093920Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4094296Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4094411Z _lazy_init(state, module) 2022-11-23T03:54:46.4094770Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4094906Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4095304Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4095428Z return func(*args, **kwargs) 2022-11-23T03:54:46.4095817Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4095912Z p_assert( 2022-11-23T03:54:46.4096253Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4096357Z traceback.print_stack() 2022-11-23T03:54:46.4096578Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4096699Z File "", line 1, in 2022-11-23T03:54:46.4096898Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4097032Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4097225Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4097367Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4097577Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4097674Z self.run() 2022-11-23T03:54:46.4097868Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4098005Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4098351Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4098476Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4098845Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4098962Z getattr(self, test_name)() 2022-11-23T03:54:46.4099329Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4099420Z fn() 2022-11-23T03:54:46.4099779Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4099897Z test(self, **param_kwargs) 2022-11-23T03:54:46.4100262Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4100380Z return func(*args, **kwargs) 2022-11-23T03:54:46.4100618Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4100723Z self.run_subtests( 2022-11-23T03:54:46.4101081Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4101235Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4101608Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4101754Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4102139Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4102316Z output = model(*input) 2022-11-23T03:54:46.4102654Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4102786Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4103172Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4103339Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4103715Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4103830Z _lazy_init(state, module) 2022-11-23T03:54:46.4104175Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4104309Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4104702Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4104823Z return func(*args, **kwargs) 2022-11-23T03:54:46.4105212Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4105309Z p_assert( 2022-11-23T03:54:46.4105650Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4105766Z traceback.print_stack() 2022-11-23T03:54:46.4105987Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4106108Z File "", line 1, in 2022-11-23T03:54:46.4106309Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4106444Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4106642Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4106788Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4106993Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4107089Z self.run() 2022-11-23T03:54:46.4107283Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4107407Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4107757Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4107882Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4108250Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4108366Z getattr(self, test_name)() 2022-11-23T03:54:46.4108736Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4108833Z fn() 2022-11-23T03:54:46.4109206Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4109328Z test(self, **param_kwargs) 2022-11-23T03:54:46.4109690Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4109808Z return func(*args, **kwargs) 2022-11-23T03:54:46.4110046Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4110150Z self.run_subtests( 2022-11-23T03:54:46.4110508Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4110662Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4111032Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4111242Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4111632Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4111730Z output = model(*input) 2022-11-23T03:54:46.4112066Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4112203Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4112590Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4112756Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4113131Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4113248Z _lazy_init(state, module) 2022-11-23T03:54:46.4113652Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4113794Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4114142Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4114260Z return func(*args, **kwargs) 2022-11-23T03:54:46.4114648Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4114744Z p_assert( 2022-11-23T03:54:46.4115089Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4115206Z traceback.print_stack() 2022-11-23T03:54:46.4115423Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4115642Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4115864Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4116072Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4116294Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4116514Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4116730Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4116948Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4117164Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4117380Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4117594Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4117812Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4118033Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4118248Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4118466Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4118683Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4118896Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4119111Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4119328Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4119545Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4119813Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4120028Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4120240Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4120461Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4120661Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4120877Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4121092Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4121307Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4121523Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4121782Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4121997Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4122212Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4122426Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4122644Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4122861Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4123081Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4123300Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4123521Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4123740Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4123955Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4124168Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4124384Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4124599Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4124811Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4125015Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4125231Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4125448Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4125559Z dist init r=1, world=2 2022-11-23T03:54:46.4125879Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4126189Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4126495Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4126804Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4127109Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4127457Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4127889Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4128197Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4128495Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4128794Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4129152Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4129454Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4129558Z dist init r=0, world=2 2022-11-23T03:54:46.4129870Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4130179Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4130486Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4130793Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4131095Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4131394Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4131693Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4131996Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4132300Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4132605Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4132905Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4133202Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4133297Z ok (9.140s) 2022-11-23T03:54:46.4133612Z test_nested_wrapped_model_offload_true_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49788 2022-11-23T03:54:46.4133822Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49789 2022-11-23T03:54:46.4134374Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.4134542Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.4134937Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.4135117Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.4135347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.4135722Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.4135875Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.4136260Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.4136490Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.4136721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.4137121Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.4137515Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.4137797Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.4138073Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.4138295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.4138511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.4138733Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4138951Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4139996Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.4140103Z warnings.warn( 2022-11-23T03:54:46.4140226Z File "", line 1, in 2022-11-23T03:54:46.4140429Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4140566Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4140761Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4140909Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4141114Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4141211Z self.run() 2022-11-23T03:54:46.4141404Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4141529Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4141878Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4142005Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4142378Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4142496Z getattr(self, test_name)() 2022-11-23T03:54:46.4142866Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4143027Z fn() 2022-11-23T03:54:46.4143411Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4143526Z test(self, **param_kwargs) 2022-11-23T03:54:46.4143889Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4144009Z return func(*args, **kwargs) 2022-11-23T03:54:46.4144248Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4144354Z self.run_subtests( 2022-11-23T03:54:46.4144713Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4144868Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4145242Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4145434Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4145825Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4145924Z output = model(*input) 2022-11-23T03:54:46.4146258Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4146395Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4146779Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4146944Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4147320Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4147434Z _lazy_init(state, module) 2022-11-23T03:54:46.4147798Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4147939Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4148286Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4148405Z return func(*args, **kwargs) 2022-11-23T03:54:46.4148792Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4148887Z p_assert( 2022-11-23T03:54:46.4149232Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4149349Z traceback.print_stack() 2022-11-23T03:54:46.4149569Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4150616Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.4150722Z warnings.warn( 2022-11-23T03:54:46.4150844Z File "", line 1, in 2022-11-23T03:54:46.4151046Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4151167Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4151361Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4151505Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4151714Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4151810Z self.run() 2022-11-23T03:54:46.4152007Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4152201Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4152554Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4152679Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4153050Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4153169Z getattr(self, test_name)() 2022-11-23T03:54:46.4153542Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4153633Z fn() 2022-11-23T03:54:46.4154008Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4154124Z test(self, **param_kwargs) 2022-11-23T03:54:46.4154489Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4154655Z return func(*args, **kwargs) 2022-11-23T03:54:46.4154881Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4154987Z self.run_subtests( 2022-11-23T03:54:46.4155350Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4155505Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4155876Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4156023Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4156406Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4156522Z output = model(*input) 2022-11-23T03:54:46.4156859Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4156994Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4157380Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4157546Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4157921Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4158036Z _lazy_init(state, module) 2022-11-23T03:54:46.4158397Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4158533Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4158879Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4158997Z return func(*args, **kwargs) 2022-11-23T03:54:46.4159384Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4159468Z p_assert( 2022-11-23T03:54:46.4159810Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4159927Z traceback.print_stack() 2022-11-23T03:54:46.4160147Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4160268Z File "", line 1, in 2022-11-23T03:54:46.4160469Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4160603Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4160795Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4160939Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4161145Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4161304Z self.run() 2022-11-23T03:54:46.4161502Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4161639Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4161993Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4162121Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4162493Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4162611Z getattr(self, test_name)() 2022-11-23T03:54:46.4162965Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4163057Z fn() 2022-11-23T03:54:46.4163432Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4163549Z test(self, **param_kwargs) 2022-11-23T03:54:46.4163958Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4164081Z return func(*args, **kwargs) 2022-11-23T03:54:46.4164319Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4164425Z self.run_subtests( 2022-11-23T03:54:46.4164788Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4164945Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4165317Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4165462Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4165846Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4165957Z output = model(*input) 2022-11-23T03:54:46.4166299Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4166432Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4166816Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4166984Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4167344Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4167458Z _lazy_init(state, module) 2022-11-23T03:54:46.4167935Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4168121Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4168737Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4168863Z return func(*args, **kwargs) 2022-11-23T03:54:46.4169252Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4169346Z p_assert( 2022-11-23T03:54:46.4169687Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4169803Z traceback.print_stack() 2022-11-23T03:54:46.4170027Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4170147Z File "", line 1, in 2022-11-23T03:54:46.4170346Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4170481Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4170677Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4170819Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4171109Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4171192Z self.run() 2022-11-23T03:54:46.4171387Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4171524Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4171877Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4172005Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4172377Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4172494Z getattr(self, test_name)() 2022-11-23T03:54:46.4172864Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4172955Z fn() 2022-11-23T03:54:46.4173327Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4173516Z test(self, **param_kwargs) 2022-11-23T03:54:46.4173890Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4174009Z return func(*args, **kwargs) 2022-11-23T03:54:46.4174247Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4174352Z self.run_subtests( 2022-11-23T03:54:46.4174714Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4174866Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4175225Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4175371Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4175762Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4175877Z output = model(*input) 2022-11-23T03:54:46.4176212Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4176347Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4176731Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4176897Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4177271Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4177386Z _lazy_init(state, module) 2022-11-23T03:54:46.4177742Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4177874Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4178228Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4178345Z return func(*args, **kwargs) 2022-11-23T03:54:46.4178732Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4178830Z p_assert( 2022-11-23T03:54:46.4179170Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4179290Z traceback.print_stack() 2022-11-23T03:54:46.4179512Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4179619Z File "", line 1, in 2022-11-23T03:54:46.4179819Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4179953Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4180150Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4180360Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4180567Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4180665Z self.run() 2022-11-23T03:54:46.4180858Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4180995Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4181347Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4181473Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4181842Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4181962Z getattr(self, test_name)() 2022-11-23T03:54:46.4182327Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4182424Z fn() 2022-11-23T03:54:46.4182849Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4182955Z test(self, **param_kwargs) 2022-11-23T03:54:46.4183323Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4183444Z return func(*args, **kwargs) 2022-11-23T03:54:46.4183681Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4183791Z self.run_subtests( 2022-11-23T03:54:46.4184150Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4184303Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4184674Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4184825Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4185212Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4185323Z output = model(*input) 2022-11-23T03:54:46.4185658Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4185796Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4186184Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4186352Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4186732Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4186846Z _lazy_init(state, module) 2022-11-23T03:54:46.4187206Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4187347Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4187678Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4187798Z return func(*args, **kwargs) 2022-11-23T03:54:46.4188185Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4188280Z p_assert( 2022-11-23T03:54:46.4188622Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4188740Z traceback.print_stack() 2022-11-23T03:54:46.4188961Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4189083Z File "", line 1, in 2022-11-23T03:54:46.4189283Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4189416Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4189670Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4189816Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4190020Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4190114Z self.run() 2022-11-23T03:54:46.4190311Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4190448Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4190797Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4190910Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4191282Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4191401Z getattr(self, test_name)() 2022-11-23T03:54:46.4191811Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4191910Z fn() 2022-11-23T03:54:46.4192288Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4192406Z test(self, **param_kwargs) 2022-11-23T03:54:46.4192775Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4192895Z return func(*args, **kwargs) 2022-11-23T03:54:46.4193132Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4193238Z self.run_subtests( 2022-11-23T03:54:46.4193595Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4193756Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4194130Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4194279Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4194661Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4194772Z output = model(*input) 2022-11-23T03:54:46.4195107Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4195228Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4195612Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4195780Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4196156Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4196272Z _lazy_init(state, module) 2022-11-23T03:54:46.4196639Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4196773Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4197118Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4197237Z return func(*args, **kwargs) 2022-11-23T03:54:46.4197623Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4197719Z p_assert( 2022-11-23T03:54:46.4198065Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4198181Z traceback.print_stack() 2022-11-23T03:54:46.4198400Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4198522Z File "", line 1, in 2022-11-23T03:54:46.4198722Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4198917Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4199097Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4199241Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4199451Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4199548Z self.run() 2022-11-23T03:54:46.4199746Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4199886Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4200239Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4200365Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4200736Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4200901Z getattr(self, test_name)() 2022-11-23T03:54:46.4201277Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4201369Z fn() 2022-11-23T03:54:46.4201742Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4201857Z test(self, **param_kwargs) 2022-11-23T03:54:46.4202222Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4202343Z return func(*args, **kwargs) 2022-11-23T03:54:46.4202580Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4202672Z self.run_subtests( 2022-11-23T03:54:46.4203033Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4203193Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4203570Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4203715Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4204097Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4204210Z output = model(*input) 2022-11-23T03:54:46.4204544Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4204678Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4205063Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4205229Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4205607Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4205722Z _lazy_init(state, module) 2022-11-23T03:54:46.4206082Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4206216Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4206566Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4206685Z return func(*args, **kwargs) 2022-11-23T03:54:46.4207070Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4207164Z p_assert( 2022-11-23T03:54:46.4207491Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4207609Z traceback.print_stack() 2022-11-23T03:54:46.4207899Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4208102Z File "", line 1, in 2022-11-23T03:54:46.4208303Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4208439Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4208636Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4208784Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4208987Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4209084Z self.run() 2022-11-23T03:54:46.4209279Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4209415Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4209771Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4209895Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4210343Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4210466Z getattr(self, test_name)() 2022-11-23T03:54:46.4210837Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4210914Z fn() 2022-11-23T03:54:46.4211293Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4211408Z test(self, **param_kwargs) 2022-11-23T03:54:46.4211778Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4211895Z return func(*args, **kwargs) 2022-11-23T03:54:46.4212130Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4212236Z self.run_subtests( 2022-11-23T03:54:46.4212600Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4212761Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4213135Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4213281Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4213666Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4213775Z output = model(*input) 2022-11-23T03:54:46.4214110Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4214244Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4214630Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4214799Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4215183Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4215284Z _lazy_init(state, module) 2022-11-23T03:54:46.4215650Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4215783Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4216131Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4216249Z return func(*args, **kwargs) 2022-11-23T03:54:46.4216636Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4216731Z p_assert( 2022-11-23T03:54:46.4217078Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4217194Z traceback.print_stack() 2022-11-23T03:54:46.4217488Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4217610Z File "", line 1, in 2022-11-23T03:54:46.4217809Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4217944Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4218138Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4218284Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4218487Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4218586Z self.run() 2022-11-23T03:54:46.4218768Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4218904Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4219258Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4219436Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4219815Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4219933Z getattr(self, test_name)() 2022-11-23T03:54:46.4220302Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4220393Z fn() 2022-11-23T03:54:46.4220767Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4220885Z test(self, **param_kwargs) 2022-11-23T03:54:46.4221255Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4221373Z return func(*args, **kwargs) 2022-11-23T03:54:46.4221609Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4221727Z self.run_subtests( 2022-11-23T03:54:46.4222087Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4222242Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4222616Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4222763Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4223134Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4223247Z output = model(*input) 2022-11-23T03:54:46.4223588Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4223725Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4224112Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4224284Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4224666Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4224781Z _lazy_init(state, module) 2022-11-23T03:54:46.4225145Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4225279Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4225625Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4225742Z return func(*args, **kwargs) 2022-11-23T03:54:46.4226128Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4226224Z p_assert( 2022-11-23T03:54:46.4226572Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4226750Z traceback.print_stack() 2022-11-23T03:54:46.4226970Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4227093Z File "", line 1, in 2022-11-23T03:54:46.4227279Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4227413Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4227605Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4227749Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4227954Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4228051Z self.run() 2022-11-23T03:54:46.4228245Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4228385Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4228789Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4228920Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4229293Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4229410Z getattr(self, test_name)() 2022-11-23T03:54:46.4229778Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4229869Z fn() 2022-11-23T03:54:46.4230243Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4230361Z test(self, **param_kwargs) 2022-11-23T03:54:46.4230709Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4230827Z return func(*args, **kwargs) 2022-11-23T03:54:46.4231071Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4231180Z self.run_subtests( 2022-11-23T03:54:46.4231540Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4231696Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4232068Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4232211Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4232597Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4232708Z output = model(*input) 2022-11-23T03:54:46.4233044Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4233178Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4233570Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4233738Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4234116Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4234230Z _lazy_init(state, module) 2022-11-23T03:54:46.4234588Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4234722Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4235068Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4235173Z return func(*args, **kwargs) 2022-11-23T03:54:46.4235561Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4235717Z p_assert( 2022-11-23T03:54:46.4236064Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4236183Z traceback.print_stack() 2022-11-23T03:54:46.4236404Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4236526Z File "", line 1, in 2022-11-23T03:54:46.4236729Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4236863Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4237055Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4237197Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4237398Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4237490Z self.run() 2022-11-23T03:54:46.4237678Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4237858Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4238207Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4238327Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4238683Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4238796Z getattr(self, test_name)() 2022-11-23T03:54:46.4239160Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4239245Z fn() 2022-11-23T03:54:46.4239609Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4239722Z test(self, **param_kwargs) 2022-11-23T03:54:46.4240079Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4240197Z return func(*args, **kwargs) 2022-11-23T03:54:46.4240431Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4240531Z self.run_subtests( 2022-11-23T03:54:46.4240885Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4241034Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4241400Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4241539Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4241919Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4242025Z output = model(*input) 2022-11-23T03:54:46.4242357Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4242489Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4242858Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4243026Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4243395Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4243506Z _lazy_init(state, module) 2022-11-23T03:54:46.4243860Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4243990Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4244329Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4244442Z return func(*args, **kwargs) 2022-11-23T03:54:46.4244825Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4244977Z p_assert( 2022-11-23T03:54:46.4245317Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4245429Z traceback.print_stack() 2022-11-23T03:54:46.4245643Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4245760Z File "", line 1, in 2022-11-23T03:54:46.4245955Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4246083Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4246272Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4246401Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4246601Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4246697Z self.run() 2022-11-23T03:54:46.4246931Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4247065Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4247411Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4247532Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4248023Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4248136Z getattr(self, test_name)() 2022-11-23T03:54:46.4248497Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4248583Z fn() 2022-11-23T03:54:46.4248949Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4249063Z test(self, **param_kwargs) 2022-11-23T03:54:46.4249430Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4249544Z return func(*args, **kwargs) 2022-11-23T03:54:46.4249773Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4249873Z self.run_subtests( 2022-11-23T03:54:46.4250217Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4250368Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4250733Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4250873Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4251251Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4251359Z output = model(*input) 2022-11-23T03:54:46.4251696Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4251825Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4252204Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4252367Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4252736Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4252845Z _lazy_init(state, module) 2022-11-23T03:54:46.4253197Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4253327Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4253667Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4253857Z return func(*args, **kwargs) 2022-11-23T03:54:46.4254243Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4254333Z p_assert( 2022-11-23T03:54:46.4254669Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4254773Z traceback.print_stack() 2022-11-23T03:54:46.4254987Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4255102Z File "", line 1, in 2022-11-23T03:54:46.4255297Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4255430Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4255616Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4255753Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4256004Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4256100Z self.run() 2022-11-23T03:54:46.4256291Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4256422Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4256766Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4256886Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4257251Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4257362Z getattr(self, test_name)() 2022-11-23T03:54:46.4257726Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4257811Z fn() 2022-11-23T03:54:46.4258171Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4258290Z test(self, **param_kwargs) 2022-11-23T03:54:46.4258649Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4258762Z return func(*args, **kwargs) 2022-11-23T03:54:46.4258994Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4259095Z self.run_subtests( 2022-11-23T03:54:46.4259448Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4259597Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4259962Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4260102Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4260485Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4260594Z output = model(*input) 2022-11-23T03:54:46.4260924Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4261051Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4261429Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4261589Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4261959Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4262069Z _lazy_init(state, module) 2022-11-23T03:54:46.4262413Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4262542Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4262949Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4263060Z return func(*args, **kwargs) 2022-11-23T03:54:46.4263442Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4263532Z p_assert( 2022-11-23T03:54:46.4263868Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4263979Z traceback.print_stack() 2022-11-23T03:54:46.4264197Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4264312Z File "", line 1, in 2022-11-23T03:54:46.4264508Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4264637Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4264827Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4265011Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4265213Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4265304Z self.run() 2022-11-23T03:54:46.4265494Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4265617Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4265962Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4266083Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4266450Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4266561Z getattr(self, test_name)() 2022-11-23T03:54:46.4266926Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4267012Z fn() 2022-11-23T03:54:46.4267388Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4267500Z test(self, **param_kwargs) 2022-11-23T03:54:46.4267862Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4267976Z return func(*args, **kwargs) 2022-11-23T03:54:46.4268209Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4268309Z self.run_subtests( 2022-11-23T03:54:46.4268661Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4268811Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4269177Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4269320Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4269702Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4269800Z output = model(*input) 2022-11-23T03:54:46.4270129Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4270258Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4270638Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4270799Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4271169Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4271278Z _lazy_init(state, module) 2022-11-23T03:54:46.4271632Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4271822Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4272167Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4272278Z return func(*args, **kwargs) 2022-11-23T03:54:46.4272658Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4272747Z p_assert( 2022-11-23T03:54:46.4273084Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4273197Z traceback.print_stack() 2022-11-23T03:54:46.4273412Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4273529Z File "", line 1, in 2022-11-23T03:54:46.4273728Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4273848Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4274085Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4274227Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4274429Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4274521Z self.run() 2022-11-23T03:54:46.4274709Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4274841Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4275188Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4275309Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4275675Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4275787Z getattr(self, test_name)() 2022-11-23T03:54:46.4276151Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4276245Z fn() 2022-11-23T03:54:46.4276611Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4276722Z test(self, **param_kwargs) 2022-11-23T03:54:46.4277084Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4277196Z return func(*args, **kwargs) 2022-11-23T03:54:46.4277419Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4277518Z self.run_subtests( 2022-11-23T03:54:46.4277870Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4278021Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4278389Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4278530Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4278907Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4279013Z output = model(*input) 2022-11-23T03:54:46.4279343Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4279471Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4279851Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4280014Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4280384Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4280493Z _lazy_init(state, module) 2022-11-23T03:54:46.4280929Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4281058Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4281395Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4281508Z return func(*args, **kwargs) 2022-11-23T03:54:46.4281880Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4281970Z p_assert( 2022-11-23T03:54:46.4282305Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4282417Z traceback.print_stack() 2022-11-23T03:54:46.4282633Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4282748Z File "", line 1, in 2022-11-23T03:54:46.4282987Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4283123Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4283311Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4283448Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4283646Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4283738Z self.run() 2022-11-23T03:54:46.4283927Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4284058Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4284402Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4284522Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4284887Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4284998Z getattr(self, test_name)() 2022-11-23T03:54:46.4285359Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4285445Z fn() 2022-11-23T03:54:46.4285813Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4285924Z test(self, **param_kwargs) 2022-11-23T03:54:46.4286284Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4286397Z return func(*args, **kwargs) 2022-11-23T03:54:46.4286631Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4286733Z self.run_subtests( 2022-11-23T03:54:46.4287086Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4287238Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4287608Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4287800Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4288179Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4288284Z output = model(*input) 2022-11-23T03:54:46.4288612Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4288743Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4289122Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4289275Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4289649Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4289832Z _lazy_init(state, module) 2022-11-23T03:54:46.4290192Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4290322Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4290660Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4290773Z return func(*args, **kwargs) 2022-11-23T03:54:46.4303664Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4303784Z p_assert( 2022-11-23T03:54:46.4304162Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4304278Z traceback.print_stack() 2022-11-23T03:54:46.4304499Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4304732Z File "", line 1, in 2022-11-23T03:54:46.4304942Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4305077Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4305264Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4305407Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4305610Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4305702Z self.run() 2022-11-23T03:54:46.4305893Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4306032Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4306388Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4306512Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4306887Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4307001Z getattr(self, test_name)() 2022-11-23T03:54:46.4307376Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4307454Z fn() 2022-11-23T03:54:46.4307830Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4307948Z test(self, **param_kwargs) 2022-11-23T03:54:46.4308310Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4308427Z return func(*args, **kwargs) 2022-11-23T03:54:46.4308664Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4308771Z self.run_subtests( 2022-11-23T03:54:46.4309134Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4309287Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4309660Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4309801Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4310184Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4310294Z output = model(*input) 2022-11-23T03:54:46.4310631Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4310762Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4311145Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4311312Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4311759Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4311859Z _lazy_init(state, module) 2022-11-23T03:54:46.4312220Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4312357Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4312702Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4312821Z return func(*args, **kwargs) 2022-11-23T03:54:46.4313205Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4313299Z p_assert( 2022-11-23T03:54:46.4313638Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4313751Z traceback.print_stack() 2022-11-23T03:54:46.4314020Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4314148Z File "", line 1, in 2022-11-23T03:54:46.4314348Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4314484Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4314677Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4314816Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4315020Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4315116Z self.run() 2022-11-23T03:54:46.4315295Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4315433Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4315781Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4315914Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4316289Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4316406Z getattr(self, test_name)() 2022-11-23T03:54:46.4316770Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4316860Z fn() 2022-11-23T03:54:46.4317231Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4317347Z test(self, **param_kwargs) 2022-11-23T03:54:46.4317712Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4317831Z return func(*args, **kwargs) 2022-11-23T03:54:46.4318069Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4318179Z self.run_subtests( 2022-11-23T03:54:46.4318535Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4318686Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4319058Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4319205Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4319573Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4319685Z output = model(*input) 2022-11-23T03:54:46.4320019Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4320152Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4320543Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4320767Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4321151Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4321263Z _lazy_init(state, module) 2022-11-23T03:54:46.4321623Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4321758Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4322103Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4322221Z return func(*args, **kwargs) 2022-11-23T03:54:46.4322606Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4322699Z p_assert( 2022-11-23T03:54:46.4323083Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4323207Z traceback.print_stack() 2022-11-23T03:54:46.4323428Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4323550Z File "", line 1, in 2022-11-23T03:54:46.4323737Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4323870Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4324060Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4324203Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4324404Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4324500Z self.run() 2022-11-23T03:54:46.4324693Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4324826Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4325180Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4325307Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4325675Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4325791Z getattr(self, test_name)() 2022-11-23T03:54:46.4326156Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4326248Z fn() 2022-11-23T03:54:46.4326616Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4326734Z test(self, **param_kwargs) 2022-11-23T03:54:46.4327097Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4327201Z return func(*args, **kwargs) 2022-11-23T03:54:46.4327442Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4327555Z self.run_subtests( 2022-11-23T03:54:46.4328368Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4328548Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4328935Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4329078Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4329458Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4329570Z output = model(*input) 2022-11-23T03:54:46.4329902Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4330035Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4330501Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4330669Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4331039Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4331147Z _lazy_init(state, module) 2022-11-23T03:54:46.4331499Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4331628Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4331964Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4332069Z return func(*args, **kwargs) 2022-11-23T03:54:46.4332448Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4332596Z p_assert( 2022-11-23T03:54:46.4332939Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4333055Z traceback.print_stack() 2022-11-23T03:54:46.4333268Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4333385Z File "", line 1, in 2022-11-23T03:54:46.4333583Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4333718Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4333907Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4334050Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4334249Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4334342Z self.run() 2022-11-23T03:54:46.4334534Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4334680Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4335028Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4335152Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4335510Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4335625Z getattr(self, test_name)() 2022-11-23T03:54:46.4335991Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4336076Z fn() 2022-11-23T03:54:46.4336438Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4336551Z test(self, **param_kwargs) 2022-11-23T03:54:46.4336911Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4337039Z return func(*args, **kwargs) 2022-11-23T03:54:46.4337281Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4337386Z self.run_subtests( 2022-11-23T03:54:46.4337743Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4337897Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4338264Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4338405Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4338788Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4338900Z output = model(*input) 2022-11-23T03:54:46.4339236Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4339432Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4339806Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4339971Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4340347Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4340459Z _lazy_init(state, module) 2022-11-23T03:54:46.4340813Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4340948Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4341294Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4341410Z return func(*args, **kwargs) 2022-11-23T03:54:46.4341839Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4341938Z p_assert( 2022-11-23T03:54:46.4342281Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4342439Z traceback.print_stack() 2022-11-23T03:54:46.4342660Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4342777Z File "", line 1, in 2022-11-23T03:54:46.4343022Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4343177Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4343407Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4343577Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4343803Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4343922Z self.run() 2022-11-23T03:54:46.4344155Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4344311Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4344730Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4344881Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4345322Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4345458Z getattr(self, test_name)() 2022-11-23T03:54:46.4345899Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4346007Z fn() 2022-11-23T03:54:46.4346456Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4346586Z test(self, **param_kwargs) 2022-11-23T03:54:46.4347035Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4347169Z return func(*args, **kwargs) 2022-11-23T03:54:46.4347452Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4347577Z self.run_subtests( 2022-11-23T03:54:46.4348002Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4348173Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4348622Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4348794Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4349258Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4349391Z output = model(*input) 2022-11-23T03:54:46.4349862Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4350019Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4350487Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4350685Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4351142Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4351273Z _lazy_init(state, module) 2022-11-23T03:54:46.4351704Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4351856Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4352273Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4352468Z return func(*args, **kwargs) 2022-11-23T03:54:46.4352937Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4353045Z p_assert( 2022-11-23T03:54:46.4353454Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4353576Z traceback.print_stack() 2022-11-23T03:54:46.4353842Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4353985Z File "", line 1, in 2022-11-23T03:54:46.4354229Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4354378Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4354606Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4354780Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4355029Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4355141Z self.run() 2022-11-23T03:54:46.4355366Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4355528Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4355943Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4356097Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4356542Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4356674Z getattr(self, test_name)() 2022-11-23T03:54:46.4357111Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4357217Z fn() 2022-11-23T03:54:46.4357652Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4357796Z test(self, **param_kwargs) 2022-11-23T03:54:46.4358230Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4358368Z return func(*args, **kwargs) 2022-11-23T03:54:46.4358652Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4358783Z self.run_subtests( 2022-11-23T03:54:46.4359187Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4359337Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4359706Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4359848Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4360231Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4360392Z output = model(*input) 2022-11-23T03:54:46.4360728Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4360861Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4361249Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4361415Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4361785Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4361896Z _lazy_init(state, module) 2022-11-23T03:54:46.4362240Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4362373Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4362783Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4362902Z return func(*args, **kwargs) 2022-11-23T03:54:46.4363289Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4363383Z p_assert( 2022-11-23T03:54:46.4363717Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4363831Z traceback.print_stack() 2022-11-23T03:54:46.4364046Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4364166Z File "", line 1, in 2022-11-23T03:54:46.4364360Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4364490Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4364679Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4364824Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4365022Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4365116Z self.run() 2022-11-23T03:54:46.4365307Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4365430Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4365777Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4365900Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4366268Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4366383Z getattr(self, test_name)() 2022-11-23T03:54:46.4366746Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4366833Z fn() 2022-11-23T03:54:46.4367209Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4367324Z test(self, **param_kwargs) 2022-11-23T03:54:46.4367684Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4367855Z return func(*args, **kwargs) 2022-11-23T03:54:46.4368092Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4368197Z self.run_subtests( 2022-11-23T03:54:46.4368558Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4368713Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4369084Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4369227Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4369717Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4369816Z output = model(*input) 2022-11-23T03:54:46.4370150Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4370283Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4370668Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4370835Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4371210Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4371323Z _lazy_init(state, module) 2022-11-23T03:54:46.4371682Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4371874Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4372224Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4372339Z return func(*args, **kwargs) 2022-11-23T03:54:46.4372720Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4372814Z p_assert( 2022-11-23T03:54:46.4373154Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4373274Z traceback.print_stack() 2022-11-23T03:54:46.4373492Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4373612Z File "", line 1, in 2022-11-23T03:54:46.4373814Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4373935Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4374135Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4374278Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4374481Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4374577Z self.run() 2022-11-23T03:54:46.4374769Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4374905Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4375252Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4375378Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4375747Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4375866Z getattr(self, test_name)() 2022-11-23T03:54:46.4376236Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4376327Z fn() 2022-11-23T03:54:46.4376699Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4376817Z test(self, **param_kwargs) 2022-11-23T03:54:46.4377178Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4377295Z return func(*args, **kwargs) 2022-11-23T03:54:46.4377521Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4377626Z self.run_subtests( 2022-11-23T03:54:46.4377983Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4378136Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4378510Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4378710Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4379097Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4379209Z output = model(*input) 2022-11-23T03:54:46.4379542Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4379677Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4380058Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4380226Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4380599Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4380710Z _lazy_init(state, module) 2022-11-23T03:54:46.4381113Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4381251Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4381595Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4381714Z return func(*args, **kwargs) 2022-11-23T03:54:46.4382083Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4382177Z p_assert( 2022-11-23T03:54:46.4382518Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4382638Z traceback.print_stack() 2022-11-23T03:54:46.4382858Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4383076Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4383298Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4383516Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4383733Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4383946Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4384158Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4384375Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4384592Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4384806Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4385025Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4385241Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4385461Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4385676Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4385888Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4386105Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4386308Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4386523Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4386738Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4386956Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4387223Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4387438Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4387652Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4387865Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4388077Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4388290Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4388502Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4388715Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4388971Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4389182Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4389397Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4389609Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4389824Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4390042Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4390255Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4390462Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4390664Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4390883Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4391100Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4391315Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4391531Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4391741Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4391952Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4392166Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4392385Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4392596Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4392820Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4392922Z dist init r=0, world=2 2022-11-23T03:54:46.4393239Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4393570Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4393885Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4394188Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4394491Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4394847Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4395151Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4395450Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4395747Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4396082Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4396387Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4396683Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4396790Z dist init r=1, world=2 2022-11-23T03:54:46.4397107Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4397415Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4397721Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4398029Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4398333Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4398632Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4398933Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4399229Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4399532Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4399830Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4400132Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4400431Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4400512Z ok (9.137s) 2022-11-23T03:54:46.4400847Z test_nested_wrapped_model_offload_true_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49941 2022-11-23T03:54:46.4401111Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49942 2022-11-23T03:54:46.4401499Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.4401663Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.4402051Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.4402227Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.4402453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.4402825Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.4402995Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.4403426Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.4403617Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.4403845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.4404245Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.4404640Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.4404921Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.4405197Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.4405414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.4405628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.4405856Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4406077Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4407130Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.4407235Z warnings.warn( 2022-11-23T03:54:46.4407355Z File "", line 1, in 2022-11-23T03:54:46.4407559Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4407680Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4407997Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4408321Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4408537Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4408631Z self.run() 2022-11-23T03:54:46.4408819Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4408953Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4409310Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4409433Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4409801Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4409913Z getattr(self, test_name)() 2022-11-23T03:54:46.4410283Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4410452Z fn() 2022-11-23T03:54:46.4410826Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4410941Z test(self, **param_kwargs) 2022-11-23T03:54:46.4411309Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4411424Z return func(*args, **kwargs) 2022-11-23T03:54:46.4411648Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4411750Z self.run_subtests( 2022-11-23T03:54:46.4412110Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4412265Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4412685Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4412837Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4413220Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4413330Z output = model(*input) 2022-11-23T03:54:46.4413662Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4413793Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4414177Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4414343Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4414717Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4414828Z _lazy_init(state, module) 2022-11-23T03:54:46.4415190Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4415320Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4415668Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4415781Z return func(*args, **kwargs) 2022-11-23T03:54:46.4416155Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4416249Z p_assert( 2022-11-23T03:54:46.4416589Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4416703Z traceback.print_stack() 2022-11-23T03:54:46.4416922Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4417970Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.4418073Z warnings.warn( 2022-11-23T03:54:46.4418193Z File "", line 1, in 2022-11-23T03:54:46.4418396Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4418524Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4418713Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4418854Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4419056Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4419148Z self.run() 2022-11-23T03:54:46.4419396Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4419530Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4419879Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4420001Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4420368Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4420480Z getattr(self, test_name)() 2022-11-23T03:54:46.4420831Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4420916Z fn() 2022-11-23T03:54:46.4421281Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4421392Z test(self, **param_kwargs) 2022-11-23T03:54:46.4421800Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4421917Z return func(*args, **kwargs) 2022-11-23T03:54:46.4422149Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4422250Z self.run_subtests( 2022-11-23T03:54:46.4422605Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4422754Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4423121Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4423260Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4423640Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4423745Z output = model(*input) 2022-11-23T03:54:46.4424080Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4424207Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4424585Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4424747Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4425109Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4425216Z _lazy_init(state, module) 2022-11-23T03:54:46.4425570Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4425700Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4426040Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4426152Z return func(*args, **kwargs) 2022-11-23T03:54:46.4426536Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4426624Z p_assert( 2022-11-23T03:54:46.4426959Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4427071Z traceback.print_stack() 2022-11-23T03:54:46.4427284Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4427399Z File "", line 1, in 2022-11-23T03:54:46.4427594Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4427722Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4427911Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4428048Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4428251Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4428402Z self.run() 2022-11-23T03:54:46.4428592Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4428723Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4429069Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4429191Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4429556Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4429667Z getattr(self, test_name)() 2022-11-23T03:54:46.4430030Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4430117Z fn() 2022-11-23T03:54:46.4430484Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4430767Z test(self, **param_kwargs) 2022-11-23T03:54:46.4431135Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4431247Z return func(*args, **kwargs) 2022-11-23T03:54:46.4431477Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4431580Z self.run_subtests( 2022-11-23T03:54:46.4431940Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4432091Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4432460Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4432591Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4432976Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4433090Z output = model(*input) 2022-11-23T03:54:46.4433423Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4433552Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4433937Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4434100Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4434473Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4434587Z _lazy_init(state, module) 2022-11-23T03:54:46.4434946Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4435079Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4435426Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4435542Z return func(*args, **kwargs) 2022-11-23T03:54:46.4435926Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4436016Z p_assert( 2022-11-23T03:54:46.4436352Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4436468Z traceback.print_stack() 2022-11-23T03:54:46.4436688Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4436795Z File "", line 1, in 2022-11-23T03:54:46.4436997Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4437128Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4437320Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4437517Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4437718Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4437816Z self.run() 2022-11-23T03:54:46.4438007Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4438142Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4438496Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4438622Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4438992Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4439106Z getattr(self, test_name)() 2022-11-23T03:54:46.4439469Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4439558Z fn() 2022-11-23T03:54:46.4439978Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4440104Z test(self, **param_kwargs) 2022-11-23T03:54:46.4440460Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4440576Z return func(*args, **kwargs) 2022-11-23T03:54:46.4440813Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4440919Z self.run_subtests( 2022-11-23T03:54:46.4441273Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4441428Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4441799Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4441941Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4442332Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4442441Z output = model(*input) 2022-11-23T03:54:46.4442769Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4442898Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4443278Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4443439Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4443809Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4443919Z _lazy_init(state, module) 2022-11-23T03:54:46.4444276Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4444408Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4444741Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4444851Z return func(*args, **kwargs) 2022-11-23T03:54:46.4445237Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4445332Z p_assert( 2022-11-23T03:54:46.4445667Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4445781Z traceback.print_stack() 2022-11-23T03:54:46.4445995Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4446114Z File "", line 1, in 2022-11-23T03:54:46.4446315Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4446447Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4446694Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4446834Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4447039Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4447133Z self.run() 2022-11-23T03:54:46.4447329Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4447464Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4447968Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4448083Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4448460Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4448577Z getattr(self, test_name)() 2022-11-23T03:54:46.4449003Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4449103Z fn() 2022-11-23T03:54:46.4449480Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4449596Z test(self, **param_kwargs) 2022-11-23T03:54:46.4449960Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4450075Z return func(*args, **kwargs) 2022-11-23T03:54:46.4450310Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4450411Z self.run_subtests( 2022-11-23T03:54:46.4450766Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4450920Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4451296Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4451440Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4451821Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4451935Z output = model(*input) 2022-11-23T03:54:46.4452269Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4452389Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4452777Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4452946Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4453321Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4453435Z _lazy_init(state, module) 2022-11-23T03:54:46.4453795Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4453932Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4454278Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4454394Z return func(*args, **kwargs) 2022-11-23T03:54:46.4454780Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4454874Z p_assert( 2022-11-23T03:54:46.4455213Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4455326Z traceback.print_stack() 2022-11-23T03:54:46.4455546Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4455662Z File "", line 1, in 2022-11-23T03:54:46.4455860Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4456057Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4456249Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4456378Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4456582Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4456678Z self.run() 2022-11-23T03:54:46.4456867Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4457001Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4457350Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4457471Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4457843Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4457957Z getattr(self, test_name)() 2022-11-23T03:54:46.4458378Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4458470Z fn() 2022-11-23T03:54:46.4458843Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4458957Z test(self, **param_kwargs) 2022-11-23T03:54:46.4459319Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4459435Z return func(*args, **kwargs) 2022-11-23T03:54:46.4459677Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4459782Z self.run_subtests( 2022-11-23T03:54:46.4460127Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4460281Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4460659Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4460803Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4461185Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4461298Z output = model(*input) 2022-11-23T03:54:46.4461633Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4461765Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4462149Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4462312Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4462686Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4462808Z _lazy_init(state, module) 2022-11-23T03:54:46.4463168Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4463303Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4463649Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4463766Z return func(*args, **kwargs) 2022-11-23T03:54:46.4464149Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4464240Z p_assert( 2022-11-23T03:54:46.4464567Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4464683Z traceback.print_stack() 2022-11-23T03:54:46.4464901Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4465021Z File "", line 1, in 2022-11-23T03:54:46.4465277Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4465414Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4465606Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4465749Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4465951Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4466045Z self.run() 2022-11-23T03:54:46.4466237Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4466372Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4466722Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4466846Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4467257Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4467377Z getattr(self, test_name)() 2022-11-23T03:54:46.4467748Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4467826Z fn() 2022-11-23T03:54:46.4468198Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4468313Z test(self, **param_kwargs) 2022-11-23T03:54:46.4468680Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4468797Z return func(*args, **kwargs) 2022-11-23T03:54:46.4469033Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4469135Z self.run_subtests( 2022-11-23T03:54:46.4469500Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4469658Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4470029Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4470172Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4470554Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4470664Z output = model(*input) 2022-11-23T03:54:46.4470994Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4471126Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4471510Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4471676Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4472059Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4472161Z _lazy_init(state, module) 2022-11-23T03:54:46.4472523Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4472657Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4473001Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4473117Z return func(*args, **kwargs) 2022-11-23T03:54:46.4473499Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4473592Z p_assert( 2022-11-23T03:54:46.4473935Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4474050Z traceback.print_stack() 2022-11-23T03:54:46.4474274Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4474454Z File "", line 1, in 2022-11-23T03:54:46.4474655Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4474792Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4474988Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4475129Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4475334Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4475428Z self.run() 2022-11-23T03:54:46.4475607Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4475744Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4476096Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4476221Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4476644Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4476765Z getattr(self, test_name)() 2022-11-23T03:54:46.4477130Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4477221Z fn() 2022-11-23T03:54:46.4477588Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4477702Z test(self, **param_kwargs) 2022-11-23T03:54:46.4478064Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4478179Z return func(*args, **kwargs) 2022-11-23T03:54:46.4478414Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4478518Z self.run_subtests( 2022-11-23T03:54:46.4478885Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4479038Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4479411Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4479552Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4479920Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4480028Z output = model(*input) 2022-11-23T03:54:46.4480361Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4480493Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4480880Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4481053Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4481424Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4481534Z _lazy_init(state, module) 2022-11-23T03:54:46.4481890Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4482019Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4482362Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4482477Z return func(*args, **kwargs) 2022-11-23T03:54:46.4482861Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4482958Z p_assert( 2022-11-23T03:54:46.4483296Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4483473Z traceback.print_stack() 2022-11-23T03:54:46.4483692Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4483813Z File "", line 1, in 2022-11-23T03:54:46.4483998Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4484133Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4484325Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4484468Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4484672Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4484767Z self.run() 2022-11-23T03:54:46.4484961Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4485095Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4485490Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4485619Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4485989Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4486102Z getattr(self, test_name)() 2022-11-23T03:54:46.4486469Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4486560Z fn() 2022-11-23T03:54:46.4486934Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4487049Z test(self, **param_kwargs) 2022-11-23T03:54:46.4487417Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4487521Z return func(*args, **kwargs) 2022-11-23T03:54:46.4488037Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4488257Z self.run_subtests( 2022-11-23T03:54:46.4488747Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4488901Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4489271Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4489413Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4489795Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4489908Z output = model(*input) 2022-11-23T03:54:46.4490240Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4490375Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4490759Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4490932Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4491306Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4491418Z _lazy_init(state, module) 2022-11-23T03:54:46.4491777Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4491909Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4492262Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4492366Z return func(*args, **kwargs) 2022-11-23T03:54:46.4492756Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4492856Z p_assert( 2022-11-23T03:54:46.4493284Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4493401Z traceback.print_stack() 2022-11-23T03:54:46.4493619Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4493741Z File "", line 1, in 2022-11-23T03:54:46.4493944Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4494076Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4494265Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4494406Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4494609Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4494704Z self.run() 2022-11-23T03:54:46.4494902Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4495103Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4495458Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4495583Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4495938Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4496054Z getattr(self, test_name)() 2022-11-23T03:54:46.4496419Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4496512Z fn() 2022-11-23T03:54:46.4496888Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4497002Z test(self, **param_kwargs) 2022-11-23T03:54:46.4497365Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4497485Z return func(*args, **kwargs) 2022-11-23T03:54:46.4497720Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4497823Z self.run_subtests( 2022-11-23T03:54:46.4498182Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4498335Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4498703Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4498848Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4499230Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4499340Z output = model(*input) 2022-11-23T03:54:46.4499674Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4499817Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4500189Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4500358Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4500733Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4500845Z _lazy_init(state, module) 2022-11-23T03:54:46.4501199Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4501333Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4501679Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4501793Z return func(*args, **kwargs) 2022-11-23T03:54:46.4502178Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4502332Z p_assert( 2022-11-23T03:54:46.4502675Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4502789Z traceback.print_stack() 2022-11-23T03:54:46.4503005Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4503125Z File "", line 1, in 2022-11-23T03:54:46.4503324Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4503453Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4503643Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4503786Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4503977Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4504072Z self.run() 2022-11-23T03:54:46.4504315Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4504452Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4504806Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4504934Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4505299Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4505415Z getattr(self, test_name)() 2022-11-23T03:54:46.4505777Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4505865Z fn() 2022-11-23T03:54:46.4506233Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4506350Z test(self, **param_kwargs) 2022-11-23T03:54:46.4506720Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4506839Z return func(*args, **kwargs) 2022-11-23T03:54:46.4507075Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4507180Z self.run_subtests( 2022-11-23T03:54:46.4507542Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4507683Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4508055Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4508202Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4508581Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4508691Z output = model(*input) 2022-11-23T03:54:46.4509030Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4509162Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4509548Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4509715Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4510085Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4510199Z _lazy_init(state, module) 2022-11-23T03:54:46.4510556Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4510691Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4511034Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4511151Z return func(*args, **kwargs) 2022-11-23T03:54:46.4511599Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4511700Z p_assert( 2022-11-23T03:54:46.4512040Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4512143Z traceback.print_stack() 2022-11-23T03:54:46.4512363Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4512486Z File "", line 1, in 2022-11-23T03:54:46.4512685Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4512818Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4513011Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4513153Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4513400Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4513505Z self.run() 2022-11-23T03:54:46.4513699Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4513833Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4514186Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4514308Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4514680Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4514797Z getattr(self, test_name)() 2022-11-23T03:54:46.4515163Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4515252Z fn() 2022-11-23T03:54:46.4515610Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4515730Z test(self, **param_kwargs) 2022-11-23T03:54:46.4516095Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4516210Z return func(*args, **kwargs) 2022-11-23T03:54:46.4516447Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4516552Z self.run_subtests( 2022-11-23T03:54:46.4516913Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4517065Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4517440Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4517586Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4517972Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4518086Z output = model(*input) 2022-11-23T03:54:46.4518426Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4518559Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4518943Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4519109Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4519482Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4519596Z _lazy_init(state, module) 2022-11-23T03:54:46.4519940Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4520074Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4520424Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4520605Z return func(*args, **kwargs) 2022-11-23T03:54:46.4520995Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4521089Z p_assert( 2022-11-23T03:54:46.4521433Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4521551Z traceback.print_stack() 2022-11-23T03:54:46.4521769Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4521890Z File "", line 1, in 2022-11-23T03:54:46.4522086Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4522218Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4522412Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4522607Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4522811Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4522908Z self.run() 2022-11-23T03:54:46.4523104Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4523227Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4523581Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4523708Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4524079Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4524192Z getattr(self, test_name)() 2022-11-23T03:54:46.4524556Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4524648Z fn() 2022-11-23T03:54:46.4525021Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4525138Z test(self, **param_kwargs) 2022-11-23T03:54:46.4525503Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4525622Z return func(*args, **kwargs) 2022-11-23T03:54:46.4525860Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4525963Z self.run_subtests( 2022-11-23T03:54:46.4526320Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4526471Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4526843Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4526989Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4527376Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4527474Z output = model(*input) 2022-11-23T03:54:46.4527862Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4527996Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4528385Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4528554Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4528928Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4529044Z _lazy_init(state, module) 2022-11-23T03:54:46.4529402Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4529601Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4529946Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4530065Z return func(*args, **kwargs) 2022-11-23T03:54:46.4530447Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4530541Z p_assert( 2022-11-23T03:54:46.4530881Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4531001Z traceback.print_stack() 2022-11-23T03:54:46.4531221Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4531340Z File "", line 1, in 2022-11-23T03:54:46.4531538Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4531659Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4531907Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4532054Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4532255Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4532352Z self.run() 2022-11-23T03:54:46.4532545Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4532681Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4533033Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4533165Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4533537Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4533653Z getattr(self, test_name)() 2022-11-23T03:54:46.4534027Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4534125Z fn() 2022-11-23T03:54:46.4534498Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4534612Z test(self, **param_kwargs) 2022-11-23T03:54:46.4534978Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4535094Z return func(*args, **kwargs) 2022-11-23T03:54:46.4535318Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4535421Z self.run_subtests( 2022-11-23T03:54:46.4535775Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4535928Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4536304Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4536450Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4536831Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4536941Z output = model(*input) 2022-11-23T03:54:46.4537271Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4537403Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4537784Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4537951Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4538329Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4538444Z _lazy_init(state, module) 2022-11-23T03:54:46.4538803Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4538996Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4539346Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4539460Z return func(*args, **kwargs) 2022-11-23T03:54:46.4539830Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4539923Z p_assert( 2022-11-23T03:54:46.4540260Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4540380Z traceback.print_stack() 2022-11-23T03:54:46.4540601Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4540719Z File "", line 1, in 2022-11-23T03:54:46.4540988Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4541127Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4541324Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4541464Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4541666Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4541764Z self.run() 2022-11-23T03:54:46.4541958Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4542091Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4542442Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4542568Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4542941Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4543043Z getattr(self, test_name)() 2022-11-23T03:54:46.4543420Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4543512Z fn() 2022-11-23T03:54:46.4543882Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4543996Z test(self, **param_kwargs) 2022-11-23T03:54:46.4544360Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4544476Z return func(*args, **kwargs) 2022-11-23T03:54:46.4544717Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4544826Z self.run_subtests( 2022-11-23T03:54:46.4545182Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4545335Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4545712Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4545858Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4546245Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4546353Z output = model(*input) 2022-11-23T03:54:46.4546683Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4546814Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4547200Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4547354Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4547732Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4547906Z _lazy_init(state, module) 2022-11-23T03:54:46.4548273Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4548409Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4548754Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4548875Z return func(*args, **kwargs) 2022-11-23T03:54:46.4549262Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4549355Z p_assert( 2022-11-23T03:54:46.4549691Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4549807Z traceback.print_stack() 2022-11-23T03:54:46.4550023Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4550196Z File "", line 1, in 2022-11-23T03:54:46.4550406Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4550542Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4550732Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4550874Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4551077Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4551160Z self.run() 2022-11-23T03:54:46.4551358Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4551496Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4551850Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4551975Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4552349Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4552466Z getattr(self, test_name)() 2022-11-23T03:54:46.4552838Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4552925Z fn() 2022-11-23T03:54:46.4553297Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4553414Z test(self, **param_kwargs) 2022-11-23T03:54:46.4553774Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4553895Z return func(*args, **kwargs) 2022-11-23T03:54:46.4554133Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4554236Z self.run_subtests( 2022-11-23T03:54:46.4554599Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4554754Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4555113Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4555254Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4555637Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4555747Z output = model(*input) 2022-11-23T03:54:46.4556079Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4556212Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4556600Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4556765Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4557204Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4557318Z _lazy_init(state, module) 2022-11-23T03:54:46.4557677Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4557810Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4558152Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4558270Z return func(*args, **kwargs) 2022-11-23T03:54:46.4558659Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4558752Z p_assert( 2022-11-23T03:54:46.4559095Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4559212Z traceback.print_stack() 2022-11-23T03:54:46.4559462Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4559592Z File "", line 1, in 2022-11-23T03:54:46.4559799Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4559933Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4560127Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4560269Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4560471Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4560566Z self.run() 2022-11-23T03:54:46.4560761Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4560897Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4561245Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4561368Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4561746Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4561862Z getattr(self, test_name)() 2022-11-23T03:54:46.4562227Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4562321Z fn() 2022-11-23T03:54:46.4562696Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4562799Z test(self, **param_kwargs) 2022-11-23T03:54:46.4563165Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4563282Z return func(*args, **kwargs) 2022-11-23T03:54:46.4563516Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4563618Z self.run_subtests( 2022-11-23T03:54:46.4563982Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4564136Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4564510Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4564651Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4565032Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4565142Z output = model(*input) 2022-11-23T03:54:46.4565474Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4565613Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4565994Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4566226Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4566609Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4566723Z _lazy_init(state, module) 2022-11-23T03:54:46.4567081Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4567214Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4567545Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4567662Z return func(*args, **kwargs) 2022-11-23T03:54:46.4568495Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4568594Z p_assert( 2022-11-23T03:54:46.4568939Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4569137Z traceback.print_stack() 2022-11-23T03:54:46.4569358Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4569478Z File "", line 1, in 2022-11-23T03:54:46.4569676Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4569808Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4570004Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4570147Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4570347Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4570446Z self.run() 2022-11-23T03:54:46.4570635Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4570771Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4571114Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4571245Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4571614Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4571727Z getattr(self, test_name)() 2022-11-23T03:54:46.4572106Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4572201Z fn() 2022-11-23T03:54:46.4572572Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4572686Z test(self, **param_kwargs) 2022-11-23T03:54:46.4573051Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4573165Z return func(*args, **kwargs) 2022-11-23T03:54:46.4573405Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4573515Z self.run_subtests( 2022-11-23T03:54:46.4573872Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4574022Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4574393Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4574538Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4574920Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4575030Z output = model(*input) 2022-11-23T03:54:46.4575350Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4575488Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4575878Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4576107Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4576485Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4576597Z _lazy_init(state, module) 2022-11-23T03:54:46.4576957Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4577093Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4577439Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4577557Z return func(*args, **kwargs) 2022-11-23T03:54:46.4577938Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4578037Z p_assert( 2022-11-23T03:54:46.4578425Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4578546Z traceback.print_stack() 2022-11-23T03:54:46.4578765Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4578887Z File "", line 1, in 2022-11-23T03:54:46.4579086Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4579221Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4579400Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4579543Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4579744Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4579839Z self.run() 2022-11-23T03:54:46.4580044Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4580188Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4580537Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4580661Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4581029Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4581145Z getattr(self, test_name)() 2022-11-23T03:54:46.4581506Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4581599Z fn() 2022-11-23T03:54:46.4581967Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4582084Z test(self, **param_kwargs) 2022-11-23T03:54:46.4582447Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4582569Z return func(*args, **kwargs) 2022-11-23T03:54:46.4582808Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4582900Z self.run_subtests( 2022-11-23T03:54:46.4583268Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4583421Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4583790Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4583940Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4584318Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4584433Z output = model(*input) 2022-11-23T03:54:46.4584766Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4584959Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4585346Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4585510Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4585884Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4585998Z _lazy_init(state, module) 2022-11-23T03:54:46.4586353Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4586491Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4586832Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4586946Z return func(*args, **kwargs) 2022-11-23T03:54:46.4587390Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4587493Z p_assert( 2022-11-23T03:54:46.4587820Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4587937Z traceback.print_stack() 2022-11-23T03:54:46.4588157Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4588275Z File "", line 1, in 2022-11-23T03:54:46.4588474Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4588608Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4588797Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4588938Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4589140Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4589233Z self.run() 2022-11-23T03:54:46.4589434Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4589571Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4589917Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4590041Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4590411Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4590524Z getattr(self, test_name)() 2022-11-23T03:54:46.4590874Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4590963Z fn() 2022-11-23T03:54:46.4591333Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4591453Z test(self, **param_kwargs) 2022-11-23T03:54:46.4591818Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4591938Z return func(*args, **kwargs) 2022-11-23T03:54:46.4592173Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4592277Z self.run_subtests( 2022-11-23T03:54:46.4592633Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4592787Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4593156Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4593301Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4593686Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4593798Z output = model(*input) 2022-11-23T03:54:46.4594194Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4594326Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4594707Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4594871Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4595248Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4595349Z _lazy_init(state, module) 2022-11-23T03:54:46.4595713Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4595849Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4596191Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4596305Z return func(*args, **kwargs) 2022-11-23T03:54:46.4596735Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4596832Z p_assert( 2022-11-23T03:54:46.4597173Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4597289Z traceback.print_stack() 2022-11-23T03:54:46.4597511Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4597631Z File "", line 1, in 2022-11-23T03:54:46.4597832Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4597965Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4598156Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4598299Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4598503Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4598588Z self.run() 2022-11-23T03:54:46.4598782Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4598917Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4599260Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4599386Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4599755Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4599869Z getattr(self, test_name)() 2022-11-23T03:54:46.4600233Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4600326Z fn() 2022-11-23T03:54:46.4600694Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4600817Z test(self, **param_kwargs) 2022-11-23T03:54:46.4601183Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4601298Z return func(*args, **kwargs) 2022-11-23T03:54:46.4601533Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4601637Z self.run_subtests( 2022-11-23T03:54:46.4601995Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4602151Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4602521Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4602655Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4603043Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4603212Z output = model(*input) 2022-11-23T03:54:46.4603555Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4603689Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4604070Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4604234Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4604608Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4604721Z _lazy_init(state, module) 2022-11-23T03:54:46.4605074Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4605210Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4605597Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4605722Z return func(*args, **kwargs) 2022-11-23T03:54:46.4606113Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4606210Z p_assert( 2022-11-23T03:54:46.4606550Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4606667Z traceback.print_stack() 2022-11-23T03:54:46.4606885Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4606992Z File "", line 1, in 2022-11-23T03:54:46.4607191Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4607329Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4607521Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4607670Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4607922Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4608018Z self.run() 2022-11-23T03:54:46.4608216Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4608355Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4608705Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4608831Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4609205Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4609321Z getattr(self, test_name)() 2022-11-23T03:54:46.4609686Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4609776Z fn() 2022-11-23T03:54:46.4610150Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4610271Z test(self, **param_kwargs) 2022-11-23T03:54:46.4610621Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4610736Z return func(*args, **kwargs) 2022-11-23T03:54:46.4610976Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4611086Z self.run_subtests( 2022-11-23T03:54:46.4611442Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4611593Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4611963Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4612108Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4612561Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4612671Z output = model(*input) 2022-11-23T03:54:46.4613001Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4613131Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4613517Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4613683Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4614060Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4614175Z _lazy_init(state, module) 2022-11-23T03:54:46.4614533Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4614732Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4615087Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4615190Z return func(*args, **kwargs) 2022-11-23T03:54:46.4615575Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4615671Z p_assert( 2022-11-23T03:54:46.4616011Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4616126Z traceback.print_stack() 2022-11-23T03:54:46.4616350Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4616467Z File "", line 1, in 2022-11-23T03:54:46.4616667Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4616801Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4616998Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4617139Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4617342Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4617435Z self.run() 2022-11-23T03:54:46.4617633Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4617768Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4618115Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4618228Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4618602Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4618723Z getattr(self, test_name)() 2022-11-23T03:54:46.4619086Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4619182Z fn() 2022-11-23T03:54:46.4619551Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4619668Z test(self, **param_kwargs) 2022-11-23T03:54:46.4620031Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4620148Z return func(*args, **kwargs) 2022-11-23T03:54:46.4620382Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4620486Z self.run_subtests( 2022-11-23T03:54:46.4620843Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4620995Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4621369Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4621582Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4621970Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4622083Z output = model(*input) 2022-11-23T03:54:46.4622417Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4622536Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4622920Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4623087Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4623464Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4623577Z _lazy_init(state, module) 2022-11-23T03:54:46.4623980Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4624122Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4624471Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4624590Z return func(*args, **kwargs) 2022-11-23T03:54:46.4624974Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4625070Z p_assert( 2022-11-23T03:54:46.4625413Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4625534Z traceback.print_stack() 2022-11-23T03:54:46.4625753Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4625872Z File "", line 1, in 2022-11-23T03:54:46.4626077Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4626211Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4626403Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4626533Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4626742Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4626836Z self.run() 2022-11-23T03:54:46.4627026Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4627162Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4627507Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4627633Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4628002Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4628119Z getattr(self, test_name)() 2022-11-23T03:54:46.4628490Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4628585Z fn() 2022-11-23T03:54:46.4628957Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4629074Z test(self, **param_kwargs) 2022-11-23T03:54:46.4629440Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4629555Z return func(*args, **kwargs) 2022-11-23T03:54:46.4629795Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 107, in test_nested_wrapped_model 2022-11-23T03:54:46.4629902Z self.run_subtests( 2022-11-23T03:54:46.4630248Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4630403Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4630840Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4630987Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4631372Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4631483Z output = model(*input) 2022-11-23T03:54:46.4631815Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4631949Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4632332Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4632501Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4632873Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4633051Z _lazy_init(state, module) 2022-11-23T03:54:46.4633419Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4633556Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4633899Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4634016Z return func(*args, **kwargs) 2022-11-23T03:54:46.4634401Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4634497Z p_assert( 2022-11-23T03:54:46.4634840Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4634944Z traceback.print_stack() 2022-11-23T03:54:46.4635167Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4635387Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4635608Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4635826Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4636045Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4636266Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4636482Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4636702Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4636916Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4637131Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4637354Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4637568Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4637784Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4637997Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4638211Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4638422Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4638636Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4638852Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4639052Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4639323Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4639538Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4639752Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4639969Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4640188Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4640402Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4640612Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4640827Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4641075Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4641301Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4641513Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4641726Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4641939Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4642155Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4642370Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4642584Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4642796Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4643015Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4643230Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4643447Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4643647Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4643864Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4644080Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4644294Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4644507Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4644726Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4644942Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4645161Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.4645264Z dist init r=0, world=2 2022-11-23T03:54:46.4645576Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4645882Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4646191Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4646494Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4646846Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4647146Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4647444Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4647849Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4648151Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4648502Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4648806Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4649102Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4649204Z dist init r=1, world=2 2022-11-23T03:54:46.4649517Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4649828Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4650134Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4650439Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4650741Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4651043Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4651342Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4651643Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4651944Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4652248Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4652551Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4652849Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4652944Z ok (8.937s) 2022-11-23T03:54:46.4653309Z test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50094 2022-11-23T03:54:46.4653564Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50095 2022-11-23T03:54:46.4653962Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.4654118Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.4654506Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.4654684Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.4654912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.4655292Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.4655456Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.4655892Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.4656075Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.4656302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.4656702Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.4657096Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.4657376Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.4657653Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.4657870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.4658090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.4658871Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4659641Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4660428Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4661210Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4661978Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4662746Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4663554Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4664322Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4665123Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4665890Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4666652Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4667419Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4668196Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4668958Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4669720Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4670483Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4671249Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4672010Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4672812Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4673571Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4674368Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4675137Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4675897Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4676660Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4677422Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4678180Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4678944Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4679704Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4680471Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4681234Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4682038Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4682799Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4683593Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4684359Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4685110Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4685866Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4686622Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4687383Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4688189Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4688946Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4689049Z dist init r=0, world=2 2022-11-23T03:54:46.4690102Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.4690207Z warnings.warn( 2022-11-23T03:54:46.4690373Z dist init r=1, world=2 2022-11-23T03:54:46.4691425Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.4691528Z warnings.warn( 2022-11-23T03:54:46.4691620Z ok (7.535s) 2022-11-23T03:54:46.4691983Z test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50247 2022-11-23T03:54:46.4692200Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50248 2022-11-23T03:54:46.4692627Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.4692802Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.4693186Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.4693364Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.4693589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.4693968Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.4694137Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.4694526Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.4694706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.4694937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.4695337Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.4695736Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.4696000Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.4696280Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.4696499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.4696718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.4697480Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4698242Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4699000Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4699771Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4700596Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4701352Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4702150Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4702913Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4703019Z dist init r=1, world=2 2022-11-23T03:54:46.4704070Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.4704173Z warnings.warn( 2022-11-23T03:54:46.4704276Z dist init r=0, world=2 2022-11-23T03:54:46.4705328Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.4705429Z warnings.warn( 2022-11-23T03:54:46.4705525Z ok (8.140s) 2022-11-23T03:54:46.4705892Z test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50400 2022-11-23T03:54:46.4706104Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50401 2022-11-23T03:54:46.4706486Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.4706653Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.4707023Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.4707201Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.4707428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.4707807Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.4707974Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.4708360Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.4708601Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.4708831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.4709232Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.4709629Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.4709903Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.4710182Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.4710399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.4710618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.4711429Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4712199Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4712954Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4713712Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4714472Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4715224Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4715981Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4716741Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4716851Z dist init r=0, world=2 2022-11-23T03:54:46.4716954Z dist init r=1, world=2 2022-11-23T03:54:46.4718002Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.4718278Z warnings.warn( 2022-11-23T03:54:46.4719320Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.4719421Z warnings.warn( 2022-11-23T03:54:46.4719515Z ok (8.531s) 2022-11-23T03:54:46.4719918Z test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50553 2022-11-23T03:54:46.4720136Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50554 2022-11-23T03:54:46.4720512Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.4720675Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.4721062Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.4721242Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.4721468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.4721843Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.4722010Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.4722405Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.4722580Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.4722803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.4723193Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.4723582Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.4723855Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.4724127Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.4724339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.4724545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.4725598Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.4725696Z warnings.warn( 2022-11-23T03:54:46.4725804Z File "", line 1, in 2022-11-23T03:54:46.4726003Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4726138Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4726330Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4726477Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4726740Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4726835Z self.run() 2022-11-23T03:54:46.4727032Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4727167Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4727519Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4727651Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4728071Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4728190Z getattr(self, test_name)() 2022-11-23T03:54:46.4728560Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4728644Z fn() 2022-11-23T03:54:46.4729075Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4729189Z test(self, **param_kwargs) 2022-11-23T03:54:46.4729559Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4729675Z return func(*args, **kwargs) 2022-11-23T03:54:46.4729958Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.4730063Z self.run_subtests( 2022-11-23T03:54:46.4730423Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4730574Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4730943Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4731090Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4731480Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4731592Z output = model(*input) 2022-11-23T03:54:46.4731927Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4732062Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4732450Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4732617Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4732995Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4733107Z _lazy_init(state, module) 2022-11-23T03:54:46.4733468Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4733610Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4733945Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4734065Z return func(*args, **kwargs) 2022-11-23T03:54:46.4734453Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4734549Z p_assert( 2022-11-23T03:54:46.4734893Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4735014Z traceback.print_stack() 2022-11-23T03:54:46.4735136Z File "", line 1, in 2022-11-23T03:54:46.4735335Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4735477Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4735671Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4735882Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4736085Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4736184Z self.run() 2022-11-23T03:54:46.4736376Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4736513Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4736867Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4736997Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4737355Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4737471Z getattr(self, test_name)() 2022-11-23T03:54:46.4737843Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4737938Z fn() 2022-11-23T03:54:46.4738353Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4738481Z test(self, **param_kwargs) 2022-11-23T03:54:46.4738851Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4738966Z return func(*args, **kwargs) 2022-11-23T03:54:46.4739244Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.4739350Z self.run_subtests( 2022-11-23T03:54:46.4739712Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4739865Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4740233Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4740383Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4740769Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4740881Z output = model(*input) 2022-11-23T03:54:46.4741218Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4741349Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4741720Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4741887Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4742271Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4742386Z _lazy_init(state, module) 2022-11-23T03:54:46.4742756Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4742896Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4743243Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4743364Z return func(*args, **kwargs) 2022-11-23T03:54:46.4743748Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4743842Z p_assert( 2022-11-23T03:54:46.4744182Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4744300Z traceback.print_stack() 2022-11-23T03:54:46.4744419Z File "", line 1, in 2022-11-23T03:54:46.4744621Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4744756Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4744948Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4745151Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4745343Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4745440Z self.run() 2022-11-23T03:54:46.4745633Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4745775Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4746131Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4746258Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4746634Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4746750Z getattr(self, test_name)() 2022-11-23T03:54:46.4747122Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4747213Z fn() 2022-11-23T03:54:46.4747637Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4747756Z test(self, **param_kwargs) 2022-11-23T03:54:46.4748121Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4748237Z return func(*args, **kwargs) 2022-11-23T03:54:46.4748521Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.4748626Z self.run_subtests( 2022-11-23T03:54:46.4748988Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4749146Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4749504Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4749653Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4750705Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.4750805Z warnings.warn( 2022-11-23T03:54:46.4750926Z File "", line 1, in 2022-11-23T03:54:46.4751127Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4751264Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4751460Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4751603Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4751807Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4751904Z self.run() 2022-11-23T03:54:46.4752095Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4752233Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4752581Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4752705Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4753079Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4753196Z getattr(self, test_name)() 2022-11-23T03:54:46.4753564Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4753655Z fn() 2022-11-23T03:54:46.4754030Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4754194Z test(self, **param_kwargs) 2022-11-23T03:54:46.4754560Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4754683Z return func(*args, **kwargs) 2022-11-23T03:54:46.4754966Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.4755074Z self.run_subtests( 2022-11-23T03:54:46.4755436Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4755589Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4755962Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4756108Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4756549Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4756664Z output = model(*input) 2022-11-23T03:54:46.4757004Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4757139Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4757523Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4757689Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4758064Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4758177Z _lazy_init(state, module) 2022-11-23T03:54:46.4758537Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4758684Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4759016Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4759137Z return func(*args, **kwargs) 2022-11-23T03:54:46.4759523Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4759617Z p_assert( 2022-11-23T03:54:46.4759955Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4760072Z traceback.print_stack() 2022-11-23T03:54:46.4760192Z File "", line 1, in 2022-11-23T03:54:46.4760393Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4760526Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4760717Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4760865Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4761072Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4761169Z self.run() 2022-11-23T03:54:46.4761361Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4761498Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4761847Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4761959Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4762335Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4762449Z getattr(self, test_name)() 2022-11-23T03:54:46.4762869Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4762978Z fn() 2022-11-23T03:54:46.4763438Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4763664Z test(self, **param_kwargs) 2022-11-23T03:54:46.4764118Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4764260Z return func(*args, **kwargs) 2022-11-23T03:54:46.4764600Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.4764724Z self.run_subtests( 2022-11-23T03:54:46.4765163Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4765348Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4765800Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4765974Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4766521Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4766655Z output = model(*input) 2022-11-23T03:54:46.4767064Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4767207Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4767673Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4768087Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4768706Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4768839Z _lazy_init(state, module) 2022-11-23T03:54:46.4769315Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4769474Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4769896Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4770041Z return func(*args, **kwargs) 2022-11-23T03:54:46.4770533Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4770646Z p_assert( 2022-11-23T03:54:46.4771067Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4771207Z traceback.print_stack() 2022-11-23T03:54:46.4771352Z File "", line 1, in 2022-11-23T03:54:46.4771594Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4771759Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4771989Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4772168Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4772411Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4772527Z self.run() 2022-11-23T03:54:46.4772761Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4772928Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4773352Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4773509Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4773979Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4774124Z getattr(self, test_name)() 2022-11-23T03:54:46.4774584Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4774697Z fn() 2022-11-23T03:54:46.4775300Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4775442Z test(self, **param_kwargs) 2022-11-23T03:54:46.4775906Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4776053Z return func(*args, **kwargs) 2022-11-23T03:54:46.4776389Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.4776528Z self.run_subtests( 2022-11-23T03:54:46.4776978Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4777163Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4777626Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4777870Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4778366Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4778498Z output = model(*input) 2022-11-23T03:54:46.4778923Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4779074Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4779556Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4779761Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4780150Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4780264Z _lazy_init(state, module) 2022-11-23T03:54:46.4780643Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4780786Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4781139Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4781258Z return func(*args, **kwargs) 2022-11-23T03:54:46.4781650Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4781744Z p_assert( 2022-11-23T03:54:46.4782084Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4782201Z traceback.print_stack() 2022-11-23T03:54:46.4782330Z File "", line 1, in 2022-11-23T03:54:46.4782528Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4782661Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4782864Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4782998Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4783203Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4783311Z self.run() 2022-11-23T03:54:46.4783509Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4783654Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4784006Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4784143Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4784513Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4784638Z getattr(self, test_name)() 2022-11-23T03:54:46.4785005Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4785104Z fn() 2022-11-23T03:54:46.4785551Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4785672Z test(self, **param_kwargs) 2022-11-23T03:54:46.4786046Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4786163Z return func(*args, **kwargs) 2022-11-23T03:54:46.4786443Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.4786557Z self.run_subtests( 2022-11-23T03:54:46.4786911Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4787072Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4787453Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4787645Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4788051Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4788162Z output = model(*input) 2022-11-23T03:54:46.4788506Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4788640Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4789033Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4789209Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4789589Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4789700Z _lazy_init(state, module) 2022-11-23T03:54:46.4790071Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4790211Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4790567Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4790680Z return func(*args, **kwargs) 2022-11-23T03:54:46.4791073Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4791168Z p_assert( 2022-11-23T03:54:46.4791518Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4791622Z traceback.print_stack() 2022-11-23T03:54:46.4791751Z File "", line 1, in 2022-11-23T03:54:46.4791952Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4792090Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4792292Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4792444Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4792654Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4792751Z self.run() 2022-11-23T03:54:46.4792944Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4793087Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4793435Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4793574Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4793959Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4794081Z getattr(self, test_name)() 2022-11-23T03:54:46.4794448Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4794611Z fn() 2022-11-23T03:54:46.4794974Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4795101Z test(self, **param_kwargs) 2022-11-23T03:54:46.4795477Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4795600Z return func(*args, **kwargs) 2022-11-23T03:54:46.4795889Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.4796005Z self.run_subtests( 2022-11-23T03:54:46.4796382Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4796532Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4796905Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4797099Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4797500Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4797614Z output = model(*input) 2022-11-23T03:54:46.4797957Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4798101Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4798490Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4798652Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4799028Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4799142Z _lazy_init(state, module) 2022-11-23T03:54:46.4799502Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4799625Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4799968Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4800085Z return func(*args, **kwargs) 2022-11-23T03:54:46.4800471Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4800565Z p_assert( 2022-11-23T03:54:46.4800905Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4801022Z traceback.print_stack() 2022-11-23T03:54:46.4801146Z File "", line 1, in 2022-11-23T03:54:46.4801345Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4801479Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4801674Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4801817Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4802023Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4802117Z self.run() 2022-11-23T03:54:46.4802309Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4802445Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4802792Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4802904Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4803271Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4803384Z getattr(self, test_name)() 2022-11-23T03:54:46.4803749Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4803896Z fn() 2022-11-23T03:54:46.4804275Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4804392Z test(self, **param_kwargs) 2022-11-23T03:54:46.4804777Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4804887Z output = model(*input) 2022-11-23T03:54:46.4805216Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4805347Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4805726Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4805891Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4806315Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4806437Z _lazy_init(state, module) 2022-11-23T03:54:46.4806797Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4806928Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4807273Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4807376Z return func(*args, **kwargs) 2022-11-23T03:54:46.4807821Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4807918Z p_assert( 2022-11-23T03:54:46.4808261Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4808378Z traceback.print_stack() 2022-11-23T03:54:46.4808498Z File "", line 1, in 2022-11-23T03:54:46.4808703Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4808840Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4809032Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4809175Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4809375Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4809470Z self.run() 2022-11-23T03:54:46.4809662Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4809803Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4810151Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4810276Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4810634Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4810760Z getattr(self, test_name)() 2022-11-23T03:54:46.4811127Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4811215Z fn() 2022-11-23T03:54:46.4811587Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4811704Z test(self, **param_kwargs) 2022-11-23T03:54:46.4812070Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4812186Z return func(*args, **kwargs) 2022-11-23T03:54:46.4812468Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.4812572Z self.run_subtests( 2022-11-23T03:54:46.4812932Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4813165Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4813537Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4813684Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4814066Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4814179Z output = model(*input) 2022-11-23T03:54:46.4814511Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4814644Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4815032Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4815184Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4815609Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4815731Z _lazy_init(state, module) 2022-11-23T03:54:46.4816092Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4816224Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4816565Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4816687Z return func(*args, **kwargs) 2022-11-23T03:54:46.4817070Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4817162Z p_assert( 2022-11-23T03:54:46.4817502Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4817616Z traceback.print_stack() 2022-11-23T03:54:46.4817736Z File "", line 1, in 2022-11-23T03:54:46.4817945Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4818078Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4818269Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4818412Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4818620Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4818703Z self.run() 2022-11-23T03:54:46.4818898Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4819035Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4819378Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4819506Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4819878Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4820002Z getattr(self, test_name)() 2022-11-23T03:54:46.4820367Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4820460Z fn() 2022-11-23T03:54:46.4820835Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4820951Z test(self, **param_kwargs) 2022-11-23T03:54:46.4821317Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4821431Z return func(*args, **kwargs) 2022-11-23T03:54:46.4821715Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.4821821Z self.run_subtests( 2022-11-23T03:54:46.4822177Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4822393Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4822770Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4822902Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4823290Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4823400Z output = model(*input) 2022-11-23T03:54:46.4823736Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4823866Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4824248Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4824413Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4824837Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4824959Z _lazy_init(state, module) 2022-11-23T03:54:46.4825323Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4825454Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4825798Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4825917Z return func(*args, **kwargs) 2022-11-23T03:54:46.4826302Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4826397Z p_assert( 2022-11-23T03:54:46.4826736Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4826853Z traceback.print_stack() 2022-11-23T03:54:46.4826975Z File "", line 1, in 2022-11-23T03:54:46.4827167Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4827301Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4827493Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4827636Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4827839Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4827932Z self.run() 2022-11-23T03:54:46.4828125Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4828267Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4828613Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4828738Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4829112Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4829230Z getattr(self, test_name)() 2022-11-23T03:54:46.4829596Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4829684Z fn() 2022-11-23T03:54:46.4830057Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4830175Z test(self, **param_kwargs) 2022-11-23T03:54:46.4830542Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4830645Z return func(*args, **kwargs) 2022-11-23T03:54:46.4830926Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.4831032Z self.run_subtests( 2022-11-23T03:54:46.4831393Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4831603Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4831976Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4832119Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4832500Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4832612Z output = model(*input) 2022-11-23T03:54:46.4832946Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4833078Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4833463Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4833625Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4834050Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4834168Z _lazy_init(state, module) 2022-11-23T03:54:46.4834532Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4834667Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4835012Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4835130Z return func(*args, **kwargs) 2022-11-23T03:54:46.4835501Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4835596Z p_assert( 2022-11-23T03:54:46.4835937Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4836054Z traceback.print_stack() 2022-11-23T03:54:46.4836186Z File "", line 1, in 2022-11-23T03:54:46.4836385Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4836517Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4836710Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4836856Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4837059Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4837157Z self.run() 2022-11-23T03:54:46.4837351Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4837488Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4837832Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4837958Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4838331Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4838437Z getattr(self, test_name)() 2022-11-23T03:54:46.4838808Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4838902Z fn() 2022-11-23T03:54:46.4839276Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4839390Z test(self, **param_kwargs) 2022-11-23T03:54:46.4839752Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4839868Z return func(*args, **kwargs) 2022-11-23T03:54:46.4840152Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.4840257Z self.run_subtests( 2022-11-23T03:54:46.4840622Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4840851Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4841228Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4841373Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4841753Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4841863Z output = model(*input) 2022-11-23T03:54:46.4842196Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4842330Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4842712Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4842879Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4843289Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4843409Z _lazy_init(state, module) 2022-11-23T03:54:46.4843771Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4843903Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4844250Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4844371Z return func(*args, **kwargs) 2022-11-23T03:54:46.4844757Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4844852Z p_assert( 2022-11-23T03:54:46.4845190Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4845307Z traceback.print_stack() 2022-11-23T03:54:46.4845431Z File "", line 1, in 2022-11-23T03:54:46.4845630Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4845766Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4845957Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4846102Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4846304Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4846403Z self.run() 2022-11-23T03:54:46.4846585Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4846720Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4847067Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4847196Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4847569Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4847753Z getattr(self, test_name)() 2022-11-23T03:54:46.4848367Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4848456Z fn() 2022-11-23T03:54:46.4848835Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4848950Z test(self, **param_kwargs) 2022-11-23T03:54:46.4849314Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4849431Z return func(*args, **kwargs) 2022-11-23T03:54:46.4849712Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.4849817Z self.run_subtests( 2022-11-23T03:54:46.4850178Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4850406Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4850786Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4850933Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4851304Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4851415Z output = model(*input) 2022-11-23T03:54:46.4851753Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4851886Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4852272Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4852487Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4852869Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4852986Z _lazy_init(state, module) 2022-11-23T03:54:46.4853341Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4853477Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4853823Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4853938Z return func(*args, **kwargs) 2022-11-23T03:54:46.4854324Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4854416Z p_assert( 2022-11-23T03:54:46.4854765Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4854887Z traceback.print_stack() 2022-11-23T03:54:46.4855006Z File "", line 1, in 2022-11-23T03:54:46.4855195Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4855326Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4855515Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4855659Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4855861Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4855955Z self.run() 2022-11-23T03:54:46.4856324Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4856442Z return func(*args, **kwargs) 2022-11-23T03:54:46.4856721Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.4856835Z self.run_subtests( 2022-11-23T03:54:46.4857191Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4857345Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4857718Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4857860Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4858248Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4858362Z output = model(*input) 2022-11-23T03:54:46.4858704Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4858836Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4859207Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4859426Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4859805Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4859918Z _lazy_init(state, module) 2022-11-23T03:54:46.4860274Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4860406Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4860751Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4860866Z return func(*args, **kwargs) 2022-11-23T03:54:46.4861251Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4861346Z p_assert( 2022-11-23T03:54:46.4861728Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4861853Z traceback.print_stack() 2022-11-23T03:54:46.4861971Z File "", line 1, in 2022-11-23T03:54:46.4862171Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4862310Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4862499Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4862642Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4862844Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4862926Z self.run() 2022-11-23T03:54:46.4863118Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4863251Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4863602Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4863730Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4864099Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4864216Z getattr(self, test_name)() 2022-11-23T03:54:46.4864585Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4864677Z fn() 2022-11-23T03:54:46.4865049Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4865166Z test(self, **param_kwargs) 2022-11-23T03:54:46.4865529Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4865645Z return func(*args, **kwargs) 2022-11-23T03:54:46.4865928Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.4866040Z self.run_subtests( 2022-11-23T03:54:46.4866402Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4866558Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4866916Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4867062Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4867445Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4867556Z output = model(*input) 2022-11-23T03:54:46.4867887Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4868020Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4868404Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4868627Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4869001Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4869122Z _lazy_init(state, module) 2022-11-23T03:54:46.4869476Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4869606Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4869950Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4870063Z return func(*args, **kwargs) 2022-11-23T03:54:46.4870451Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4870546Z p_assert( 2022-11-23T03:54:46.4870937Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4871059Z traceback.print_stack() 2022-11-23T03:54:46.4871167Z File "", line 1, in 2022-11-23T03:54:46.4871369Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4871501Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4871690Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4871829Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4872032Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4872125Z self.run() 2022-11-23T03:54:46.4872317Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4872452Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4872803Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4872935Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4873304Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4873418Z getattr(self, test_name)() 2022-11-23T03:54:46.4873783Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4873873Z fn() 2022-11-23T03:54:46.4874244Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4874360Z test(self, **param_kwargs) 2022-11-23T03:54:46.4874713Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4874828Z return func(*args, **kwargs) 2022-11-23T03:54:46.4875114Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.4875222Z self.run_subtests( 2022-11-23T03:54:46.4875582Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4875735Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4876111Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4876253Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4876641Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4876751Z output = model(*input) 2022-11-23T03:54:46.4877082Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4877215Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4877604Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4877819Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4878196Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4878312Z _lazy_init(state, module) 2022-11-23T03:54:46.4878673Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4878805Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4879150Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4879254Z return func(*args, **kwargs) 2022-11-23T03:54:46.4879638Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4879733Z p_assert( 2022-11-23T03:54:46.4880120Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4880241Z traceback.print_stack() 2022-11-23T03:54:46.4880362Z File "", line 1, in 2022-11-23T03:54:46.4880563Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4880695Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4880886Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4881027Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4881227Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4881326Z self.run() 2022-11-23T03:54:46.4881522Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4881658Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4882010Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4882136Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4882493Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4882617Z getattr(self, test_name)() 2022-11-23T03:54:46.4882982Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4883072Z fn() 2022-11-23T03:54:46.4883442Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4883559Z test(self, **param_kwargs) 2022-11-23T03:54:46.4883922Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4884040Z return func(*args, **kwargs) 2022-11-23T03:54:46.4884325Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.4884435Z self.run_subtests( 2022-11-23T03:54:46.4884791Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4884943Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4885311Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4885453Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4885833Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4885944Z output = model(*input) 2022-11-23T03:54:46.4886279Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4886413Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4886858Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4887011Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4887384Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4887498Z _lazy_init(state, module) 2022-11-23T03:54:46.4887987Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4888127Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4888474Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4888591Z return func(*args, **kwargs) 2022-11-23T03:54:46.4888977Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4889075Z p_assert( 2022-11-23T03:54:46.4889480Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4889602Z traceback.print_stack() 2022-11-23T03:54:46.4889728Z File "", line 1, in 2022-11-23T03:54:46.4889922Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4890058Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4890247Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4890389Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4890592Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4890675Z self.run() 2022-11-23T03:54:46.4890868Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4891003Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4891361Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4891487Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4891861Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4891973Z getattr(self, test_name)() 2022-11-23T03:54:46.4892341Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4892431Z fn() 2022-11-23T03:54:46.4892803Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4892917Z test(self, **param_kwargs) 2022-11-23T03:54:46.4893282Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4893400Z return func(*args, **kwargs) 2022-11-23T03:54:46.4893683Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.4893788Z self.run_subtests( 2022-11-23T03:54:46.4894146Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4894302Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4894672Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4894804Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4895186Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4895298Z output = model(*input) 2022-11-23T03:54:46.4895629Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4895762Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4896219Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4896381Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4896753Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4896866Z _lazy_init(state, module) 2022-11-23T03:54:46.4897222Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4897356Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4897700Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4897819Z return func(*args, **kwargs) 2022-11-23T03:54:46.4898205Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4898380Z p_assert( 2022-11-23T03:54:46.4898731Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4898851Z traceback.print_stack() 2022-11-23T03:54:46.4898973Z File "", line 1, in 2022-11-23T03:54:46.4899158Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4899293Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4899486Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4899629Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4899833Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4899930Z self.run() 2022-11-23T03:54:46.4900122Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4900257Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4900613Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4900739Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4901112Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4901229Z getattr(self, test_name)() 2022-11-23T03:54:46.4901593Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4901685Z fn() 2022-11-23T03:54:46.4902061Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4902180Z test(self, **param_kwargs) 2022-11-23T03:54:46.4902545Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4902650Z return func(*args, **kwargs) 2022-11-23T03:54:46.4902935Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.4903046Z self.run_subtests( 2022-11-23T03:54:46.4903405Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4903560Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4903931Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4904075Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4904461Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4904570Z output = model(*input) 2022-11-23T03:54:46.4904901Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4905034Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4905482Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4905648Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4906023Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4906137Z _lazy_init(state, module) 2022-11-23T03:54:46.4906494Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4906630Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4906977Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4907080Z return func(*args, **kwargs) 2022-11-23T03:54:46.4907275Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4907475Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4907825Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4907951Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4908322Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4908437Z getattr(self, test_name)() 2022-11-23T03:54:46.4908801Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4908891Z fn() 2022-11-23T03:54:46.4909264Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4909379Z test(self, **param_kwargs) 2022-11-23T03:54:46.4909745Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4909868Z return func(*args, **kwargs) 2022-11-23T03:54:46.4910147Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.4910253Z self.run_subtests( 2022-11-23T03:54:46.4910610Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4910761Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4911131Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4911278Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4911648Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4911756Z output = model(*input) 2022-11-23T03:54:46.4912093Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4912225Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4912609Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4912773Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4913148Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4913263Z _lazy_init(state, module) 2022-11-23T03:54:46.4913623Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4913757Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4914105Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4914221Z return func(*args, **kwargs) 2022-11-23T03:54:46.4914608Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4914758Z p_assert( 2022-11-23T03:54:46.4915105Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4915220Z traceback.print_stack() 2022-11-23T03:54:46.4915338Z File "", line 1, in 2022-11-23T03:54:46.4915525Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4915658Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4915850Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4915993Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4916195Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4916289Z self.run() 2022-11-23T03:54:46.4916486Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4916678Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4917029Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4917156Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4917526Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4917642Z getattr(self, test_name)() 2022-11-23T03:54:46.4918013Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4918100Z fn() 2022-11-23T03:54:46.4918476Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4918593Z test(self, **param_kwargs) 2022-11-23T03:54:46.4918961Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4919071Z return func(*args, **kwargs) 2022-11-23T03:54:46.4919353Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.4919457Z self.run_subtests( 2022-11-23T03:54:46.4919818Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4919970Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4920337Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4920483Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4920865Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4920977Z output = model(*input) 2022-11-23T03:54:46.4921310Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4921445Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4921830Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4921995Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4922371Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4922481Z _lazy_init(state, module) 2022-11-23T03:54:46.4922837Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4922973Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4923318Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4923435Z return func(*args, **kwargs) 2022-11-23T03:54:46.4923867Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4923961Z p_assert( 2022-11-23T03:54:46.4924305Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4924419Z traceback.print_stack() 2022-11-23T03:54:46.4924539Z File "", line 1, in 2022-11-23T03:54:46.4924739Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4924874Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4925065Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4925209Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4925408Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4925504Z self.run() 2022-11-23T03:54:46.4925740Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4925883Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4926231Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4926357Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4926724Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4926842Z getattr(self, test_name)() 2022-11-23T03:54:46.4927196Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4927286Z fn() 2022-11-23T03:54:46.4927662Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4927831Z test(self, **param_kwargs) 2022-11-23T03:54:46.4928203Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4928321Z return func(*args, **kwargs) 2022-11-23T03:54:46.4928605Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.4928709Z self.run_subtests( 2022-11-23T03:54:46.4929070Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4929225Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4929596Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4929739Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4930121Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4930230Z output = model(*input) 2022-11-23T03:54:46.4930569Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4930704Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4931088Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4931254Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4931614Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4931727Z _lazy_init(state, module) 2022-11-23T03:54:46.4932081Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4932214Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4932556Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4932673Z return func(*args, **kwargs) 2022-11-23T03:54:46.4933513Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4934287Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4935056Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4935874Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4936645Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4937421Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4938191Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4938956Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4939714Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4940479Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4941235Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4941985Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4942792Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4943553Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.4943661Z dist init r=0, world=2 2022-11-23T03:54:46.4943978Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4944326Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4944636Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4944936Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4945238Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4945540Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4945838Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4946146Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4946445Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4946742Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4947042Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4947339Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.4947737Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4947841Z p_assert( 2022-11-23T03:54:46.4948185Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4948303Z traceback.print_stack() 2022-11-23T03:54:46.4948427Z File "", line 1, in 2022-11-23T03:54:46.4948614Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4948750Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4948941Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4949083Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4949286Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4949380Z self.run() 2022-11-23T03:54:46.4949573Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4949759Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4950108Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4950231Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4950601Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4950718Z getattr(self, test_name)() 2022-11-23T03:54:46.4951086Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4951174Z fn() 2022-11-23T03:54:46.4951548Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4951668Z test(self, **param_kwargs) 2022-11-23T03:54:46.4952031Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4952187Z return func(*args, **kwargs) 2022-11-23T03:54:46.4952476Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.4952584Z self.run_subtests( 2022-11-23T03:54:46.4952946Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4953097Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4953465Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4953611Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4953994Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4954103Z output = model(*input) 2022-11-23T03:54:46.4954442Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4954580Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4954968Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4955135Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4955512Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4955626Z _lazy_init(state, module) 2022-11-23T03:54:46.4955988Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4956126Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4956474Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4956591Z return func(*args, **kwargs) 2022-11-23T03:54:46.4956970Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4957062Z p_assert( 2022-11-23T03:54:46.4957405Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4957521Z traceback.print_stack() 2022-11-23T03:54:46.4957625Z dist init r=1, world=2 2022-11-23T03:54:46.4957942Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4958250Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4958553Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4958913Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4959213Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4959514Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4959813Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4960108Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4960447Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4960749Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4961045Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4961342Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.4961740Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4961833Z p_assert( 2022-11-23T03:54:46.4962174Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4962294Z traceback.print_stack() 2022-11-23T03:54:46.4962413Z File "", line 1, in 2022-11-23T03:54:46.4962611Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4962746Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4962938Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4963080Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4963270Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4963368Z self.run() 2022-11-23T03:54:46.4963560Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4963698Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4964044Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4964169Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4964545Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4964662Z getattr(self, test_name)() 2022-11-23T03:54:46.4965033Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4965128Z fn() 2022-11-23T03:54:46.4965504Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4965617Z test(self, **param_kwargs) 2022-11-23T03:54:46.4965983Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4966100Z return func(*args, **kwargs) 2022-11-23T03:54:46.4966377Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.4966481Z self.run_subtests( 2022-11-23T03:54:46.4966900Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4967041Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4967413Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4967560Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4967999Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4968111Z output = model(*input) 2022-11-23T03:54:46.4968443Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4968574Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4968954Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4969178Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4969564Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4969684Z _lazy_init(state, module) 2022-11-23T03:54:46.4970044Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4970179Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4970521Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4970639Z return func(*args, **kwargs) 2022-11-23T03:54:46.4971025Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4971119Z p_assert( 2022-11-23T03:54:46.4971459Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4971586Z traceback.print_stack() 2022-11-23T03:54:46.4971667Z ok (8.141s) 2022-11-23T03:54:46.4972026Z test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50706 2022-11-23T03:54:46.4972238Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50707 2022-11-23T03:54:46.4972614Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.4972777Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.4973166Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.4973345Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.4973572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.4973948Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.4974114Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.4974501Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.4974681Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.4974919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.4975315Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.4975706Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.4975986Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.4976345Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.4976562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.4976781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.4977834Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.4977936Z warnings.warn( 2022-11-23T03:54:46.4978057Z File "", line 1, in 2022-11-23T03:54:46.4978300Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4978427Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4978624Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4978768Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4978989Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4979084Z self.run() 2022-11-23T03:54:46.4979280Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4979415Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4979767Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4979895Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4980268Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4980388Z getattr(self, test_name)() 2022-11-23T03:54:46.4980760Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4980860Z fn() 2022-11-23T03:54:46.4981232Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4981347Z test(self, **param_kwargs) 2022-11-23T03:54:46.4981714Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4981832Z return func(*args, **kwargs) 2022-11-23T03:54:46.4982111Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.4982203Z self.run_subtests( 2022-11-23T03:54:46.4982567Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4982727Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4983102Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4983254Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4983635Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4983743Z output = model(*input) 2022-11-23T03:54:46.4984079Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4984211Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4984600Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4984764Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4985144Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4985308Z _lazy_init(state, module) 2022-11-23T03:54:46.4985672Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4985808Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4986155Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4986276Z return func(*args, **kwargs) 2022-11-23T03:54:46.4986664Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4986745Z p_assert( 2022-11-23T03:54:46.4987086Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4987203Z traceback.print_stack() 2022-11-23T03:54:46.4987323Z File "", line 1, in 2022-11-23T03:54:46.4987573Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4987712Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4987904Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4988050Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4988259Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4988355Z self.run() 2022-11-23T03:54:46.4988546Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4988681Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4989027Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4989150Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4989521Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4989643Z getattr(self, test_name)() 2022-11-23T03:54:46.4990014Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4990091Z fn() 2022-11-23T03:54:46.4990481Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4990599Z test(self, **param_kwargs) 2022-11-23T03:54:46.4990966Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.4991084Z return func(*args, **kwargs) 2022-11-23T03:54:46.4991363Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.4991470Z self.run_subtests( 2022-11-23T03:54:46.4991825Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.4991986Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.4992355Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.4992496Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.4992882Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.4993000Z output = model(*input) 2022-11-23T03:54:46.4993343Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.4993478Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.4993863Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.4994028Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.4994462Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.4994577Z _lazy_init(state, module) 2022-11-23T03:54:46.4994920Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.4995056Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.4995401Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.4995520Z return func(*args, **kwargs) 2022-11-23T03:54:46.4995920Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.4996015Z p_assert( 2022-11-23T03:54:46.4996355Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.4996468Z traceback.print_stack() 2022-11-23T03:54:46.4996587Z File "", line 1, in 2022-11-23T03:54:46.4996833Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.4996967Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.4997158Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.4997297Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.4997499Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.4997595Z self.run() 2022-11-23T03:54:46.4997789Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.4997912Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.4998264Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.4998394Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.4998768Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.4998889Z getattr(self, test_name)() 2022-11-23T03:54:46.4999252Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.4999340Z fn() 2022-11-23T03:54:46.4999713Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.4999832Z test(self, **param_kwargs) 2022-11-23T03:54:46.5000196Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5000313Z return func(*args, **kwargs) 2022-11-23T03:54:46.5000598Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5000701Z self.run_subtests( 2022-11-23T03:54:46.5001060Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5001218Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5001592Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5001740Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5002788Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.5002891Z warnings.warn( 2022-11-23T03:54:46.5003013Z File "", line 1, in 2022-11-23T03:54:46.5003222Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5003397Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5003592Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5003734Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5003938Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5004034Z self.run() 2022-11-23T03:54:46.5004228Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5004360Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5004711Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5004838Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5005211Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5005487Z getattr(self, test_name)() 2022-11-23T03:54:46.5005859Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5005951Z fn() 2022-11-23T03:54:46.5006320Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5006436Z test(self, **param_kwargs) 2022-11-23T03:54:46.5006800Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5006917Z return func(*args, **kwargs) 2022-11-23T03:54:46.5007186Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5007294Z self.run_subtests( 2022-11-23T03:54:46.5007651Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5007930Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5008309Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5008456Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5008840Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5008952Z output = model(*input) 2022-11-23T03:54:46.5009290Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5009421Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5009807Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5009971Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5010348Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5010466Z _lazy_init(state, module) 2022-11-23T03:54:46.5010823Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5010959Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5011304Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5011424Z return func(*args, **kwargs) 2022-11-23T03:54:46.5011814Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5011896Z p_assert( 2022-11-23T03:54:46.5012239Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5012358Z traceback.print_stack() 2022-11-23T03:54:46.5012475Z File "", line 1, in 2022-11-23T03:54:46.5012748Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5012881Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5013069Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5013210Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5013414Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5013507Z self.run() 2022-11-23T03:54:46.5013695Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5013831Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5014184Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5014312Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5014682Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5014852Z getattr(self, test_name)() 2022-11-23T03:54:46.5015217Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5015307Z fn() 2022-11-23T03:54:46.5015680Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5015795Z test(self, **param_kwargs) 2022-11-23T03:54:46.5016159Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5016275Z return func(*args, **kwargs) 2022-11-23T03:54:46.5016556Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5016668Z self.run_subtests( 2022-11-23T03:54:46.5017025Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5017190Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5017571Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5017718Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5018100Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5018213Z output = model(*input) 2022-11-23T03:54:46.5018551Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5018682Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5019070Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5019240Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5019624Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5019727Z _lazy_init(state, module) 2022-11-23T03:54:46.5020087Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5020221Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5020570Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5020688Z return func(*args, **kwargs) 2022-11-23T03:54:46.5021074Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5021170Z p_assert( 2022-11-23T03:54:46.5021508Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5021624Z traceback.print_stack() 2022-11-23T03:54:46.5021743Z File "", line 1, in 2022-11-23T03:54:46.5022003Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5022139Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5022330Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5022474Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5022679Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5022777Z self.run() 2022-11-23T03:54:46.5022955Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5023092Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5023444Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5023571Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5023982Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5024102Z getattr(self, test_name)() 2022-11-23T03:54:46.5024472Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5024562Z fn() 2022-11-23T03:54:46.5024930Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5025049Z test(self, **param_kwargs) 2022-11-23T03:54:46.5025411Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5025527Z return func(*args, **kwargs) 2022-11-23T03:54:46.5025808Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5025912Z self.run_subtests( 2022-11-23T03:54:46.5026271Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5026426Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5026795Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5026941Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5027324Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5027422Z output = model(*input) 2022-11-23T03:54:46.5027758Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5027891Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5028272Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5028440Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5028816Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5028929Z _lazy_init(state, module) 2022-11-23T03:54:46.5029288Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5029419Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5029764Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5029885Z return func(*args, **kwargs) 2022-11-23T03:54:46.5030270Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5030363Z p_assert( 2022-11-23T03:54:46.5030705Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5030820Z traceback.print_stack() 2022-11-23T03:54:46.5031001Z File "", line 1, in 2022-11-23T03:54:46.5031201Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5031322Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5031518Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5031659Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5031863Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5031957Z self.run() 2022-11-23T03:54:46.5032150Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5032283Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5032635Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5032760Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5033175Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5033296Z getattr(self, test_name)() 2022-11-23T03:54:46.5033663Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5033752Z fn() 2022-11-23T03:54:46.5034122Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5034242Z test(self, **param_kwargs) 2022-11-23T03:54:46.5034605Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5034723Z return func(*args, **kwargs) 2022-11-23T03:54:46.5034990Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5035101Z self.run_subtests( 2022-11-23T03:54:46.5035463Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5035619Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5035986Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5036131Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5036512Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5036623Z output = model(*input) 2022-11-23T03:54:46.5036957Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5037091Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5037478Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5037640Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5038021Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5038137Z _lazy_init(state, module) 2022-11-23T03:54:46.5038496Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5038628Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5038974Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5039088Z return func(*args, **kwargs) 2022-11-23T03:54:46.5039473Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5039554Z p_assert( 2022-11-23T03:54:46.5039897Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5040015Z traceback.print_stack() 2022-11-23T03:54:46.5040195Z File "", line 1, in 2022-11-23T03:54:46.5040397Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5040536Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5040725Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5040866Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5041072Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5041168Z self.run() 2022-11-23T03:54:46.5041357Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5041494Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5041843Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5041966Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5042394Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5042523Z getattr(self, test_name)() 2022-11-23T03:54:46.5042890Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5042985Z fn() 2022-11-23T03:54:46.5043344Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5043460Z test(self, **param_kwargs) 2022-11-23T03:54:46.5043824Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5043940Z return func(*args, **kwargs) 2022-11-23T03:54:46.5044221Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5044326Z self.run_subtests( 2022-11-23T03:54:46.5044693Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5044851Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5045222Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5045369Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5045751Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5045861Z output = model(*input) 2022-11-23T03:54:46.5046198Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5046333Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5046719Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5046888Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5047266Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5047379Z _lazy_init(state, module) 2022-11-23T03:54:46.5047781Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5047902Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5048249Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5048368Z return func(*args, **kwargs) 2022-11-23T03:54:46.5048752Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5048845Z p_assert( 2022-11-23T03:54:46.5049186Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5049373Z traceback.print_stack() 2022-11-23T03:54:46.5049497Z File "", line 1, in 2022-11-23T03:54:46.5049694Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5049829Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5050022Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5050163Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5050365Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5050461Z self.run() 2022-11-23T03:54:46.5050654Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5050793Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5051130Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5051256Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5051682Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5051803Z getattr(self, test_name)() 2022-11-23T03:54:46.5052169Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5052260Z fn() 2022-11-23T03:54:46.5052632Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5052749Z test(self, **param_kwargs) 2022-11-23T03:54:46.5053129Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5053244Z output = model(*input) 2022-11-23T03:54:46.5053577Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5053712Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5054104Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5054273Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5054650Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5054764Z _lazy_init(state, module) 2022-11-23T03:54:46.5055123Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5055259Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5055604Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5055709Z return func(*args, **kwargs) 2022-11-23T03:54:46.5056092Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5056186Z p_assert( 2022-11-23T03:54:46.5056530Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5056649Z traceback.print_stack() 2022-11-23T03:54:46.5056773Z File "", line 1, in 2022-11-23T03:54:46.5056971Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5057104Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5057295Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5057435Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5057640Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5057737Z self.run() 2022-11-23T03:54:46.5057930Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5058068Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5058418Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5058603Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5058963Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5059080Z getattr(self, test_name)() 2022-11-23T03:54:46.5059447Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5059539Z fn() 2022-11-23T03:54:46.5059912Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5060033Z test(self, **param_kwargs) 2022-11-23T03:54:46.5060398Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5060519Z return func(*args, **kwargs) 2022-11-23T03:54:46.5060843Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5060957Z self.run_subtests( 2022-11-23T03:54:46.5061323Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5061479Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5061850Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5061994Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5062380Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5062492Z output = model(*input) 2022-11-23T03:54:46.5062828Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5062960Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5063339Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5063504Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5063879Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5063992Z _lazy_init(state, module) 2022-11-23T03:54:46.5064350Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5064486Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5064832Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5064949Z return func(*args, **kwargs) 2022-11-23T03:54:46.5065334Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5065436Z p_assert( 2022-11-23T03:54:46.5065778Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5065898Z traceback.print_stack() 2022-11-23T03:54:46.5066019Z File "", line 1, in 2022-11-23T03:54:46.5066215Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5066352Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5066544Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5066686Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5066889Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5066972Z self.run() 2022-11-23T03:54:46.5067166Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5067303Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5067707Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5067833Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5068210Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5068330Z getattr(self, test_name)() 2022-11-23T03:54:46.5068695Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5068789Z fn() 2022-11-23T03:54:46.5069158Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5069275Z test(self, **param_kwargs) 2022-11-23T03:54:46.5069643Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5069760Z return func(*args, **kwargs) 2022-11-23T03:54:46.5070080Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5070195Z self.run_subtests( 2022-11-23T03:54:46.5070562Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5070719Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5071077Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5071227Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5071610Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5071722Z output = model(*input) 2022-11-23T03:54:46.5072058Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5072193Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5072587Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5072751Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5073124Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5073236Z _lazy_init(state, module) 2022-11-23T03:54:46.5073595Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5073730Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5074073Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5074190Z return func(*args, **kwargs) 2022-11-23T03:54:46.5074576Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5074676Z p_assert( 2022-11-23T03:54:46.5075022Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5075139Z traceback.print_stack() 2022-11-23T03:54:46.5075247Z File "", line 1, in 2022-11-23T03:54:46.5075446Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5075586Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5075779Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5075926Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5076135Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5076232Z self.run() 2022-11-23T03:54:46.5076424Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5076562Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5077012Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5077141Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5077513Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5077630Z getattr(self, test_name)() 2022-11-23T03:54:46.5078001Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5078092Z fn() 2022-11-23T03:54:46.5078466Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5078582Z test(self, **param_kwargs) 2022-11-23T03:54:46.5078934Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5079052Z return func(*args, **kwargs) 2022-11-23T03:54:46.5079388Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5079498Z self.run_subtests( 2022-11-23T03:54:46.5079865Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5080018Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5080394Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5080537Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5080924Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5081034Z output = model(*input) 2022-11-23T03:54:46.5081369Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5081509Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5081895Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5082065Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5082440Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5082554Z _lazy_init(state, module) 2022-11-23T03:54:46.5082917Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5083050Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5083394Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5083498Z return func(*args, **kwargs) 2022-11-23T03:54:46.5083889Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5083990Z p_assert( 2022-11-23T03:54:46.5084329Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5084447Z traceback.print_stack() 2022-11-23T03:54:46.5084568Z File "", line 1, in 2022-11-23T03:54:46.5084768Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5084902Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5085094Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5085238Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5085440Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5085537Z self.run() 2022-11-23T03:54:46.5085731Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5085870Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5086278Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5086406Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5086777Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5086880Z getattr(self, test_name)() 2022-11-23T03:54:46.5087250Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5087340Z fn() 2022-11-23T03:54:46.5087824Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5087943Z test(self, **param_kwargs) 2022-11-23T03:54:46.5088310Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5088426Z return func(*args, **kwargs) 2022-11-23T03:54:46.5088779Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5088888Z self.run_subtests( 2022-11-23T03:54:46.5089254Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5089412Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5089784Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5089928Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5090310Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5090422Z output = model(*input) 2022-11-23T03:54:46.5090755Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5090896Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5091281Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5091434Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5091808Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5091923Z _lazy_init(state, module) 2022-11-23T03:54:46.5092280Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5092416Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5092761Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5092879Z return func(*args, **kwargs) 2022-11-23T03:54:46.5093268Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5093366Z p_assert( 2022-11-23T03:54:46.5093707Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5093822Z traceback.print_stack() 2022-11-23T03:54:46.5093945Z File "", line 1, in 2022-11-23T03:54:46.5094144Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5094277Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5094472Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5094613Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5094817Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5094900Z self.run() 2022-11-23T03:54:46.5095094Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5095359Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5095710Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5095836Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5096207Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5096324Z getattr(self, test_name)() 2022-11-23T03:54:46.5096691Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5096783Z fn() 2022-11-23T03:54:46.5097162Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5097279Z test(self, **param_kwargs) 2022-11-23T03:54:46.5097643Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5097812Z return func(*args, **kwargs) 2022-11-23T03:54:46.5098091Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5098196Z self.run_subtests( 2022-11-23T03:54:46.5098561Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5098715Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5099085Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5099217Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5099598Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5099709Z output = model(*input) 2022-11-23T03:54:46.5100047Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5100183Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5100562Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5100728Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5101102Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5101220Z _lazy_init(state, module) 2022-11-23T03:54:46.5101577Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5101710Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5102054Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5102171Z return func(*args, **kwargs) 2022-11-23T03:54:46.5102560Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5102657Z p_assert( 2022-11-23T03:54:46.5103000Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5103118Z traceback.print_stack() 2022-11-23T03:54:46.5103240Z File "", line 1, in 2022-11-23T03:54:46.5103426Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5103560Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5103750Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5103896Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5104100Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5104196Z self.run() 2022-11-23T03:54:46.5104569Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5104750Z return func(*args, **kwargs) 2022-11-23T03:54:46.5105032Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5105138Z self.run_subtests( 2022-11-23T03:54:46.5105499Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5105657Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5106028Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5106174Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5106555Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5106666Z output = model(*input) 2022-11-23T03:54:46.5107044Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5107182Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5107553Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5107719Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5108097Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5108215Z _lazy_init(state, module) 2022-11-23T03:54:46.5108573Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5108708Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5109058Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5109177Z return func(*args, **kwargs) 2022-11-23T03:54:46.5109565Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5109662Z p_assert( 2022-11-23T03:54:46.5110003Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5110120Z traceback.print_stack() 2022-11-23T03:54:46.5110240Z File "", line 1, in 2022-11-23T03:54:46.5110439Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5110571Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5110762Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5110906Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5111094Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5111191Z self.run() 2022-11-23T03:54:46.5111393Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5111528Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5111878Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5112008Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5112380Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5112495Z getattr(self, test_name)() 2022-11-23T03:54:46.5112859Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5112949Z fn() 2022-11-23T03:54:46.5113325Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5113441Z test(self, **param_kwargs) 2022-11-23T03:54:46.5113810Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5114009Z return func(*args, **kwargs) 2022-11-23T03:54:46.5114291Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5114397Z self.run_subtests( 2022-11-23T03:54:46.5114759Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5114913Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5115270Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5115416Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5115799Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5115911Z output = model(*input) 2022-11-23T03:54:46.5116298Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5116436Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5116824Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5116990Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5117368Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5117482Z _lazy_init(state, module) 2022-11-23T03:54:46.5117841Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5117977Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5118323Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5118448Z return func(*args, **kwargs) 2022-11-23T03:54:46.5118834Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5118928Z p_assert( 2022-11-23T03:54:46.5119272Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5119390Z traceback.print_stack() 2022-11-23T03:54:46.5119497Z File "", line 1, in 2022-11-23T03:54:46.5119695Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5119830Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5120020Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5120163Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5120367Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5120467Z self.run() 2022-11-23T03:54:46.5120664Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5120802Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5121147Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5121276Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5121646Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5121764Z getattr(self, test_name)() 2022-11-23T03:54:46.5122135Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5122225Z fn() 2022-11-23T03:54:46.5122596Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5122714Z test(self, **param_kwargs) 2022-11-23T03:54:46.5123135Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5123255Z return func(*args, **kwargs) 2022-11-23T03:54:46.5123538Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5123643Z self.run_subtests( 2022-11-23T03:54:46.5124002Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5124156Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5124532Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5124677Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5125062Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5125223Z output = model(*input) 2022-11-23T03:54:46.5125565Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5125698Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5126081Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5126248Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5126625Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5126741Z _lazy_init(state, module) 2022-11-23T03:54:46.5127098Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5127233Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5127581Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5127745Z return func(*args, **kwargs) 2022-11-23T03:54:46.5128132Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5128227Z p_assert( 2022-11-23T03:54:46.5128569Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5128686Z traceback.print_stack() 2022-11-23T03:54:46.5128806Z File "", line 1, in 2022-11-23T03:54:46.5129006Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5129141Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5129336Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5129483Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5129686Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5129790Z self.run() 2022-11-23T03:54:46.5129987Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5130122Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5130470Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5130596Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5130954Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5131070Z getattr(self, test_name)() 2022-11-23T03:54:46.5131438Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5131529Z fn() 2022-11-23T03:54:46.5131904Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5132022Z test(self, **param_kwargs) 2022-11-23T03:54:46.5132467Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5132584Z return func(*args, **kwargs) 2022-11-23T03:54:46.5132864Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5132969Z self.run_subtests( 2022-11-23T03:54:46.5133331Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5133487Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5133862Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5134009Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5134391Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5134557Z output = model(*input) 2022-11-23T03:54:46.5134899Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5135032Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5135403Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5135572Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5135947Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5136061Z _lazy_init(state, module) 2022-11-23T03:54:46.5136420Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5136553Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5136902Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5137024Z return func(*args, **kwargs) 2022-11-23T03:54:46.5137412Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5137507Z p_assert( 2022-11-23T03:54:46.5137849Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5137966Z traceback.print_stack() 2022-11-23T03:54:46.5138088Z File "", line 1, in 2022-11-23T03:54:46.5138288Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5138423Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5138615Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5138762Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5138972Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5139058Z self.run() 2022-11-23T03:54:46.5139253Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5139389Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5139736Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5139862Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5140236Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5140356Z getattr(self, test_name)() 2022-11-23T03:54:46.5140723Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5140817Z fn() 2022-11-23T03:54:46.5141189Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5141369Z test(self, **param_kwargs) 2022-11-23T03:54:46.5141738Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5141855Z return func(*args, **kwargs) 2022-11-23T03:54:46.5142140Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5142246Z self.run_subtests( 2022-11-23T03:54:46.5142605Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5142758Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5143116Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5143262Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5143685Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5143803Z output = model(*input) 2022-11-23T03:54:46.5144141Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5144281Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5144666Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5144834Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5145210Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5145324Z _lazy_init(state, module) 2022-11-23T03:54:46.5145686Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5145820Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5146169Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5146290Z return func(*args, **kwargs) 2022-11-23T03:54:46.5146677Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5146772Z p_assert( 2022-11-23T03:54:46.5147113Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5147230Z traceback.print_stack() 2022-11-23T03:54:46.5147337Z File "", line 1, in 2022-11-23T03:54:46.5147538Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5147673Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5147866Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5148010Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5148218Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5148318Z self.run() 2022-11-23T03:54:46.5148510Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5148646Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5148994Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5149120Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5149493Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5149611Z getattr(self, test_name)() 2022-11-23T03:54:46.5149976Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5150067Z fn() 2022-11-23T03:54:46.5150440Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5150623Z test(self, **param_kwargs) 2022-11-23T03:54:46.5150976Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5151094Z return func(*args, **kwargs) 2022-11-23T03:54:46.5151375Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5151484Z self.run_subtests( 2022-11-23T03:54:46.5151845Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5151999Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5152372Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5152518Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5152946Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5153064Z output = model(*input) 2022-11-23T03:54:46.5153407Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5153545Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5153931Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5154096Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5154471Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5154583Z _lazy_init(state, module) 2022-11-23T03:54:46.5154944Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5155079Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5155430Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5155535Z return func(*args, **kwargs) 2022-11-23T03:54:46.5155733Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5155871Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5156215Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5156340Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5156709Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5156831Z getattr(self, test_name)() 2022-11-23T03:54:46.5157199Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5157291Z fn() 2022-11-23T03:54:46.5157667Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5157786Z test(self, **param_kwargs) 2022-11-23T03:54:46.5158152Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5158269Z return func(*args, **kwargs) 2022-11-23T03:54:46.5158549Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5158654Z self.run_subtests( 2022-11-23T03:54:46.5159014Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5159168Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5159540Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5159671Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5160128Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5160243Z output = model(*input) 2022-11-23T03:54:46.5160577Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5160710Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5161096Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5161265Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5161640Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5161754Z _lazy_init(state, module) 2022-11-23T03:54:46.5162115Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5162299Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5162652Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5162769Z return func(*args, **kwargs) 2022-11-23T03:54:46.5163160Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5163255Z p_assert( 2022-11-23T03:54:46.5163597Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5163715Z traceback.print_stack() 2022-11-23T03:54:46.5163838Z File "", line 1, in 2022-11-23T03:54:46.5164023Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5164157Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5164358Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5164509Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5164713Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5164813Z self.run() 2022-11-23T03:54:46.5165010Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5165147Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5165494Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5165620Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5165994Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5166115Z getattr(self, test_name)() 2022-11-23T03:54:46.5166482Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5166576Z fn() 2022-11-23T03:54:46.5166952Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5167073Z test(self, **param_kwargs) 2022-11-23T03:54:46.5167441Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5167544Z return func(*args, **kwargs) 2022-11-23T03:54:46.5167931Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5168038Z self.run_subtests( 2022-11-23T03:54:46.5168402Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5168556Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5168928Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5169072Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5169532Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5169647Z output = model(*input) 2022-11-23T03:54:46.5169984Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5170116Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5170504Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5170670Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5171050Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5171164Z _lazy_init(state, module) 2022-11-23T03:54:46.5171525Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5171724Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5172074Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5172194Z return func(*args, **kwargs) 2022-11-23T03:54:46.5172565Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5172664Z p_assert( 2022-11-23T03:54:46.5173005Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5173121Z traceback.print_stack() 2022-11-23T03:54:46.5173242Z File "", line 1, in 2022-11-23T03:54:46.5173445Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5173579Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5173772Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5173923Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5174126Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5174223Z self.run() 2022-11-23T03:54:46.5174417Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5174554Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5174901Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5175026Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5175397Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5175500Z getattr(self, test_name)() 2022-11-23T03:54:46.5175868Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5175960Z fn() 2022-11-23T03:54:46.5176340Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5176458Z test(self, **param_kwargs) 2022-11-23T03:54:46.5176823Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5176941Z return func(*args, **kwargs) 2022-11-23T03:54:46.5177221Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5177326Z self.run_subtests( 2022-11-23T03:54:46.5177686Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5177840Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5178213Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5178423Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5178809Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5178922Z output = model(*input) 2022-11-23T03:54:46.5179257Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5179390Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5179776Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5179941Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5180304Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5180419Z _lazy_init(state, module) 2022-11-23T03:54:46.5180834Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5180976Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5181322Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5181441Z return func(*args, **kwargs) 2022-11-23T03:54:46.5182215Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.5182991Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.5183767Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.5184535Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.5185298Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.5186070Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.5186844Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.5187605Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.5188367Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.5189184Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.5189942Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.5190745Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.5190855Z dist init r=0, world=2 2022-11-23T03:54:46.5191171Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.5191482Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.5191787Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.5192095Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.5192401Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.5192704Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.5193007Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.5193311Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.5193614Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.5193919Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.5194220Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.5194517Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.5194913Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5195012Z p_assert( 2022-11-23T03:54:46.5195355Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5195460Z traceback.print_stack() 2022-11-23T03:54:46.5195633Z File "", line 1, in 2022-11-23T03:54:46.5195836Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5195972Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5196167Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5196311Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5196518Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5196615Z self.run() 2022-11-23T03:54:46.5196809Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5196946Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5197299Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5197426Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5197841Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5197967Z getattr(self, test_name)() 2022-11-23T03:54:46.5198342Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5198433Z fn() 2022-11-23T03:54:46.5198794Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5198910Z test(self, **param_kwargs) 2022-11-23T03:54:46.5199276Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5199392Z return func(*args, **kwargs) 2022-11-23T03:54:46.5199676Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5199781Z self.run_subtests( 2022-11-23T03:54:46.5200150Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5200307Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5200685Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5200829Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5201216Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5201327Z output = model(*input) 2022-11-23T03:54:46.5201662Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5201798Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5202184Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5202354Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5202736Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5202851Z _lazy_init(state, module) 2022-11-23T03:54:46.5203211Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5203332Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5203679Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5203797Z return func(*args, **kwargs) 2022-11-23T03:54:46.5204188Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5204282Z p_assert( 2022-11-23T03:54:46.5204624Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5204743Z traceback.print_stack() 2022-11-23T03:54:46.5204910Z dist init r=1, world=2 2022-11-23T03:54:46.5205229Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.5205540Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.5205849Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.5206154Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.5206462Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.5206812Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.5207120Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.5207419Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.5207763Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.5208064Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.5208370Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.5208672Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.5209073Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5209168Z p_assert( 2022-11-23T03:54:46.5209510Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5209630Z traceback.print_stack() 2022-11-23T03:54:46.5209752Z File "", line 1, in 2022-11-23T03:54:46.5209954Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5210074Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5210273Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5210422Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5210630Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5210727Z self.run() 2022-11-23T03:54:46.5210929Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5211068Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5211418Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5211544Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5211924Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5212040Z getattr(self, test_name)() 2022-11-23T03:54:46.5212411Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5212570Z fn() 2022-11-23T03:54:46.5212952Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5213071Z test(self, **param_kwargs) 2022-11-23T03:54:46.5213443Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5213560Z return func(*args, **kwargs) 2022-11-23T03:54:46.5213829Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5213936Z self.run_subtests( 2022-11-23T03:54:46.5214297Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5214452Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5214825Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5215026Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5215419Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5215531Z output = model(*input) 2022-11-23T03:54:46.5215866Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5215999Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5216384Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5216550Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5216926Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5217041Z _lazy_init(state, module) 2022-11-23T03:54:46.5217406Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5217542Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5217886Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5218003Z return func(*args, **kwargs) 2022-11-23T03:54:46.5218390Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5218471Z p_assert( 2022-11-23T03:54:46.5218809Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5218927Z traceback.print_stack() 2022-11-23T03:54:46.5219021Z ok (8.236s) 2022-11-23T03:54:46.5219394Z test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50859 2022-11-23T03:54:46.5219610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50860 2022-11-23T03:54:46.5219986Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.5220154Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.5220542Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.5220725Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.5220955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.5221328Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.5221496Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.5221884Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.5222134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.5222370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.5222774Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.5223171Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.5223452Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.5223732Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.5223952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.5224158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.5225279Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.5225390Z warnings.warn( 2022-11-23T03:54:46.5225499Z File "", line 1, in 2022-11-23T03:54:46.5225701Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5225836Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5226028Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5226173Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5226380Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5226480Z self.run() 2022-11-23T03:54:46.5226673Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5226810Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5227161Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5227289Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5227660Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5227779Z getattr(self, test_name)() 2022-11-23T03:54:46.5228148Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5228239Z fn() 2022-11-23T03:54:46.5228614Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5228738Z test(self, **param_kwargs) 2022-11-23T03:54:46.5229091Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5229208Z return func(*args, **kwargs) 2022-11-23T03:54:46.5229490Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5229598Z self.run_subtests( 2022-11-23T03:54:46.5229959Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5230112Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5230485Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5230630Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5231017Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5231190Z output = model(*input) 2022-11-23T03:54:46.5231531Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5231669Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5232058Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5232224Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5232604Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5232719Z _lazy_init(state, module) 2022-11-23T03:54:46.5233078Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5233213Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5233607Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5233713Z return func(*args, **kwargs) 2022-11-23T03:54:46.5234103Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5234199Z p_assert( 2022-11-23T03:54:46.5234542Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5234664Z traceback.print_stack() 2022-11-23T03:54:46.5234788Z File "", line 1, in 2022-11-23T03:54:46.5234987Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5235123Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5235315Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5235460Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5235671Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5235769Z self.run() 2022-11-23T03:54:46.5235963Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5236103Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5236449Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5236576Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5236933Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5237055Z getattr(self, test_name)() 2022-11-23T03:54:46.5237422Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5237514Z fn() 2022-11-23T03:54:46.5237897Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5238021Z test(self, **param_kwargs) 2022-11-23T03:54:46.5238386Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5238504Z return func(*args, **kwargs) 2022-11-23T03:54:46.5238788Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5238892Z self.run_subtests( 2022-11-23T03:54:46.5239255Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5239411Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5239783Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5239926Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5240379Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5240492Z output = model(*input) 2022-11-23T03:54:46.5240826Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5240959Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5241344Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5241497Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5241875Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5241989Z _lazy_init(state, module) 2022-11-23T03:54:46.5242356Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5242492Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5242887Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5243014Z return func(*args, **kwargs) 2022-11-23T03:54:46.5243405Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5243500Z p_assert( 2022-11-23T03:54:46.5243843Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5243961Z traceback.print_stack() 2022-11-23T03:54:46.5244086Z File "", line 1, in 2022-11-23T03:54:46.5244287Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5244421Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5244616Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5244759Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5244972Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5245055Z self.run() 2022-11-23T03:54:46.5245250Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5245388Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5245736Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5245863Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5246240Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5246357Z getattr(self, test_name)() 2022-11-23T03:54:46.5246723Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5246813Z fn() 2022-11-23T03:54:46.5247191Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5247311Z test(self, **param_kwargs) 2022-11-23T03:54:46.5247675Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5247838Z return func(*args, **kwargs) 2022-11-23T03:54:46.5248122Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5248228Z self.run_subtests( 2022-11-23T03:54:46.5248589Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5248747Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5249119Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5249250Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5250405Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.5250520Z warnings.warn( 2022-11-23T03:54:46.5250627Z File "", line 1, in 2022-11-23T03:54:46.5250829Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5250964Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5251156Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5251304Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5251574Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5251684Z self.run() 2022-11-23T03:54:46.5251878Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5252013Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5252365Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5252493Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5252865Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5252984Z getattr(self, test_name)() 2022-11-23T03:54:46.5253354Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5253446Z fn() 2022-11-23T03:54:46.5253824Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5253934Z test(self, **param_kwargs) 2022-11-23T03:54:46.5254300Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5254418Z return func(*args, **kwargs) 2022-11-23T03:54:46.5254703Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5254812Z self.run_subtests( 2022-11-23T03:54:46.5255171Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5255325Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5255698Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5255843Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5256230Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5256345Z output = model(*input) 2022-11-23T03:54:46.5256680Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5256815Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5257201Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5257369Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5257748Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5257863Z _lazy_init(state, module) 2022-11-23T03:54:46.5258223Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5258362Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5258765Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5258887Z return func(*args, **kwargs) 2022-11-23T03:54:46.5259276Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5259372Z p_assert( 2022-11-23T03:54:46.5259713Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5259831Z traceback.print_stack() 2022-11-23T03:54:46.5259951Z File "", line 1, in 2022-11-23T03:54:46.5260152Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5260287Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5260479Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5260623Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5260875Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5260974Z self.run() 2022-11-23T03:54:46.5261169Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5261306Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5261656Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5261784Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5262140Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5262259Z getattr(self, test_name)() 2022-11-23T03:54:46.5262626Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5262718Z fn() 2022-11-23T03:54:46.5263093Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5263218Z test(self, **param_kwargs) 2022-11-23T03:54:46.5263581Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5263698Z return func(*args, **kwargs) 2022-11-23T03:54:46.5263982Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5264089Z self.run_subtests( 2022-11-23T03:54:46.5264448Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5264602Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5264972Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5265118Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5265506Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5265622Z output = model(*input) 2022-11-23T03:54:46.5265957Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5266093Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5266465Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5266635Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5267009Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5267124Z _lazy_init(state, module) 2022-11-23T03:54:46.5267483Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5267618Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5268032Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5268151Z return func(*args, **kwargs) 2022-11-23T03:54:46.5268537Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5268633Z p_assert( 2022-11-23T03:54:46.5268975Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5269091Z traceback.print_stack() 2022-11-23T03:54:46.5269212Z File "", line 1, in 2022-11-23T03:54:46.5269412Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5269548Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5269741Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5269885Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5270122Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5270222Z self.run() 2022-11-23T03:54:46.5270419Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5270556Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5270907Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5271034Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5271406Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5271528Z getattr(self, test_name)() 2022-11-23T03:54:46.5271894Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5271983Z fn() 2022-11-23T03:54:46.5272362Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5272483Z test(self, **param_kwargs) 2022-11-23T03:54:46.5272849Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5272967Z return func(*args, **kwargs) 2022-11-23T03:54:46.5273248Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5273352Z self.run_subtests( 2022-11-23T03:54:46.5273714Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5273872Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5274230Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5274378Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5274768Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5274880Z output = model(*input) 2022-11-23T03:54:46.5275213Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5275347Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5275735Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5275907Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5276283Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5276398Z _lazy_init(state, module) 2022-11-23T03:54:46.5276756Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5276961Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5277310Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5277427Z return func(*args, **kwargs) 2022-11-23T03:54:46.5277817Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5277913Z p_assert( 2022-11-23T03:54:46.5278254Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5278372Z traceback.print_stack() 2022-11-23T03:54:46.5278481Z File "", line 1, in 2022-11-23T03:54:46.5278682Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5278818Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5279011Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5279201Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5279412Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5279509Z self.run() 2022-11-23T03:54:46.5279703Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5279840Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5280194Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5280323Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5280693Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5280810Z getattr(self, test_name)() 2022-11-23T03:54:46.5281182Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5281273Z fn() 2022-11-23T03:54:46.5281652Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5281774Z test(self, **param_kwargs) 2022-11-23T03:54:46.5282124Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5282241Z return func(*args, **kwargs) 2022-11-23T03:54:46.5282521Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5282630Z self.run_subtests( 2022-11-23T03:54:46.5282990Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5283146Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5283517Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5283663Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5284053Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5284170Z output = model(*input) 2022-11-23T03:54:46.5284505Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5284639Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5285027Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5285193Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5285570Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5285686Z _lazy_init(state, module) 2022-11-23T03:54:46.5286047Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5286257Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5286608Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5286712Z return func(*args, **kwargs) 2022-11-23T03:54:46.5287099Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5287196Z p_assert( 2022-11-23T03:54:46.5287538Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5287654Z traceback.print_stack() 2022-11-23T03:54:46.5287888Z File "", line 1, in 2022-11-23T03:54:46.5288088Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5288222Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5288415Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5288629Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5288838Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5288935Z self.run() 2022-11-23T03:54:46.5289129Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5289265Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5289615Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5289736Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5290092Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5290204Z getattr(self, test_name)() 2022-11-23T03:54:46.5290566Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5290652Z fn() 2022-11-23T03:54:46.5291028Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5291140Z test(self, **param_kwargs) 2022-11-23T03:54:46.5291498Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5291609Z return func(*args, **kwargs) 2022-11-23T03:54:46.5291885Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5291986Z self.run_subtests( 2022-11-23T03:54:46.5292340Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5292488Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5292856Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5293000Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5293376Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5293483Z output = model(*input) 2022-11-23T03:54:46.5293812Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5293940Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5294322Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5294475Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5294846Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5294955Z _lazy_init(state, module) 2022-11-23T03:54:46.5295313Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5295628Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5295973Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5296085Z return func(*args, **kwargs) 2022-11-23T03:54:46.5296467Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5296558Z p_assert( 2022-11-23T03:54:46.5296893Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5297004Z traceback.print_stack() 2022-11-23T03:54:46.5297124Z File "", line 1, in 2022-11-23T03:54:46.5297319Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5297448Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5297636Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5297828Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5298033Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5298116Z self.run() 2022-11-23T03:54:46.5298306Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5298437Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5298780Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5298900Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5299265Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5299378Z getattr(self, test_name)() 2022-11-23T03:54:46.5299739Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5299825Z fn() 2022-11-23T03:54:46.5300202Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5300313Z test(self, **param_kwargs) 2022-11-23T03:54:46.5300691Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5300797Z output = model(*input) 2022-11-23T03:54:46.5301125Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5301252Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5301632Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5301792Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5302162Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5302271Z _lazy_init(state, module) 2022-11-23T03:54:46.5302625Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5302753Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5303093Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5303205Z return func(*args, **kwargs) 2022-11-23T03:54:46.5303587Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5303676Z p_assert( 2022-11-23T03:54:46.5304012Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5304123Z traceback.print_stack() 2022-11-23T03:54:46.5304239Z File "", line 1, in 2022-11-23T03:54:46.5304433Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5304624Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5304812Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5304950Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5305153Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5305243Z self.run() 2022-11-23T03:54:46.5305424Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5305556Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5305901Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5306022Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5306388Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5306500Z getattr(self, test_name)() 2022-11-23T03:54:46.5306912Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5307001Z fn() 2022-11-23T03:54:46.5307371Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5307483Z test(self, **param_kwargs) 2022-11-23T03:54:46.5307844Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5307955Z return func(*args, **kwargs) 2022-11-23T03:54:46.5308232Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5308333Z self.run_subtests( 2022-11-23T03:54:46.5308687Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5308834Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5309206Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5309345Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5309715Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5309823Z output = model(*input) 2022-11-23T03:54:46.5310152Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5310281Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5310659Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5310820Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5311190Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5311303Z _lazy_init(state, module) 2022-11-23T03:54:46.5311658Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5311787Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5312128Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5312239Z return func(*args, **kwargs) 2022-11-23T03:54:46.5312621Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5312711Z p_assert( 2022-11-23T03:54:46.5313047Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5313159Z traceback.print_stack() 2022-11-23T03:54:46.5313276Z File "", line 1, in 2022-11-23T03:54:46.5313475Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5313661Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5313847Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5313985Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5314184Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5314276Z self.run() 2022-11-23T03:54:46.5314464Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5314595Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5314937Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5315057Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5315422Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5315534Z getattr(self, test_name)() 2022-11-23T03:54:46.5315962Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5316051Z fn() 2022-11-23T03:54:46.5316423Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5316535Z test(self, **param_kwargs) 2022-11-23T03:54:46.5316894Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5317006Z return func(*args, **kwargs) 2022-11-23T03:54:46.5317269Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5317370Z self.run_subtests( 2022-11-23T03:54:46.5317723Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5317875Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5318245Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5318385Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5318761Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5318866Z output = model(*input) 2022-11-23T03:54:46.5319196Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5319325Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5319700Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5319863Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5320239Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5320352Z _lazy_init(state, module) 2022-11-23T03:54:46.5320707Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5320836Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5321176Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5321289Z return func(*args, **kwargs) 2022-11-23T03:54:46.5321670Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5321752Z p_assert( 2022-11-23T03:54:46.5322089Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5322201Z traceback.print_stack() 2022-11-23T03:54:46.5322318Z File "", line 1, in 2022-11-23T03:54:46.5322514Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5322710Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5322898Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5323035Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5323235Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5323326Z self.run() 2022-11-23T03:54:46.5323517Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5323648Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5323992Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5324114Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5324481Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5324644Z getattr(self, test_name)() 2022-11-23T03:54:46.5325003Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5325089Z fn() 2022-11-23T03:54:46.5325458Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5325569Z test(self, **param_kwargs) 2022-11-23T03:54:46.5325931Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5326043Z return func(*args, **kwargs) 2022-11-23T03:54:46.5326318Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5326421Z self.run_subtests( 2022-11-23T03:54:46.5326774Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5326929Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5327297Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5327439Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5327872Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5327979Z output = model(*input) 2022-11-23T03:54:46.5328308Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5328439Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5328823Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5328984Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5329357Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5329461Z _lazy_init(state, module) 2022-11-23T03:54:46.5329816Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5329945Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5330285Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5330397Z return func(*args, **kwargs) 2022-11-23T03:54:46.5330779Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5330868Z p_assert( 2022-11-23T03:54:46.5331205Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5331317Z traceback.print_stack() 2022-11-23T03:54:46.5331433Z File "", line 1, in 2022-11-23T03:54:46.5331701Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5331831Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5332017Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5332155Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5332354Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5332446Z self.run() 2022-11-23T03:54:46.5332634Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5332758Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5333105Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5333226Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5333590Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5333758Z getattr(self, test_name)() 2022-11-23T03:54:46.5334131Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5334216Z fn() 2022-11-23T03:54:46.5334583Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5334696Z test(self, **param_kwargs) 2022-11-23T03:54:46.5335057Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5335170Z return func(*args, **kwargs) 2022-11-23T03:54:46.5335444Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5335546Z self.run_subtests( 2022-11-23T03:54:46.5335901Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5336056Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5336421Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5336559Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5336940Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5337039Z output = model(*input) 2022-11-23T03:54:46.5337370Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5337500Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5337879Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5338040Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5338419Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5338532Z _lazy_init(state, module) 2022-11-23T03:54:46.5338887Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5339017Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5339356Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5339470Z return func(*args, **kwargs) 2022-11-23T03:54:46.5339853Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5339944Z p_assert( 2022-11-23T03:54:46.5340279Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5340392Z traceback.print_stack() 2022-11-23T03:54:46.5340508Z File "", line 1, in 2022-11-23T03:54:46.5340766Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5340888Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5341077Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5341215Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5341416Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5341507Z self.run() 2022-11-23T03:54:46.5341696Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5341829Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5342175Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5342297Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5342706Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5342828Z getattr(self, test_name)() 2022-11-23T03:54:46.5343194Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5343283Z fn() 2022-11-23T03:54:46.5343651Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5343763Z test(self, **param_kwargs) 2022-11-23T03:54:46.5344121Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5344233Z return func(*args, **kwargs) 2022-11-23T03:54:46.5344508Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5344604Z self.run_subtests( 2022-11-23T03:54:46.5344964Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5345116Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5345482Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5345623Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5346001Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5346107Z output = model(*input) 2022-11-23T03:54:46.5346437Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5346565Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5346946Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5347107Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5347483Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5347592Z _lazy_init(state, module) 2022-11-23T03:54:46.5347947Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5348078Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5348420Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5348533Z return func(*args, **kwargs) 2022-11-23T03:54:46.5348914Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5348995Z p_assert( 2022-11-23T03:54:46.5349331Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5349443Z traceback.print_stack() 2022-11-23T03:54:46.5349625Z File "", line 1, in 2022-11-23T03:54:46.5349823Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5358506Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5358732Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5358874Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5359076Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5359171Z self.run() 2022-11-23T03:54:46.5359590Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5359706Z return func(*args, **kwargs) 2022-11-23T03:54:46.5359984Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5360087Z self.run_subtests( 2022-11-23T03:54:46.5360553Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5360714Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5361078Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5361222Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5361605Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5361713Z output = model(*input) 2022-11-23T03:54:46.5362044Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5362172Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5362561Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5362732Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5363106Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5363218Z _lazy_init(state, module) 2022-11-23T03:54:46.5363574Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5363708Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5364053Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5364171Z return func(*args, **kwargs) 2022-11-23T03:54:46.5364559Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5364651Z p_assert( 2022-11-23T03:54:46.5364990Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5365113Z traceback.print_stack() 2022-11-23T03:54:46.5365221Z File "", line 1, in 2022-11-23T03:54:46.5365419Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5365556Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5365748Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5365888Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5366092Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5366187Z self.run() 2022-11-23T03:54:46.5366379Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5366512Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5366860Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5366984Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5367425Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5367542Z getattr(self, test_name)() 2022-11-23T03:54:46.5368086Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5368177Z fn() 2022-11-23T03:54:46.5368554Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5368672Z test(self, **param_kwargs) 2022-11-23T03:54:46.5369022Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5369137Z return func(*args, **kwargs) 2022-11-23T03:54:46.5369423Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5369527Z self.run_subtests( 2022-11-23T03:54:46.5369953Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5370110Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5370484Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5370629Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5371012Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5371124Z output = model(*input) 2022-11-23T03:54:46.5371456Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5371590Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5371976Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5372149Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5372526Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5372640Z _lazy_init(state, module) 2022-11-23T03:54:46.5372997Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5373128Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5373476Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5373581Z return func(*args, **kwargs) 2022-11-23T03:54:46.5373968Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5374065Z p_assert( 2022-11-23T03:54:46.5374405Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5374525Z traceback.print_stack() 2022-11-23T03:54:46.5374650Z File "", line 1, in 2022-11-23T03:54:46.5374848Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5374984Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5375181Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5375330Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5375534Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5375630Z self.run() 2022-11-23T03:54:46.5375824Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5375962Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5376310Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5376440Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5376867Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5376983Z getattr(self, test_name)() 2022-11-23T03:54:46.5377347Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5377439Z fn() 2022-11-23T03:54:46.5377812Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5377933Z test(self, **param_kwargs) 2022-11-23T03:54:46.5378296Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5378410Z return func(*args, **kwargs) 2022-11-23T03:54:46.5378694Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5378848Z self.run_subtests( 2022-11-23T03:54:46.5379216Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5379370Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5379740Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5379885Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5380269Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5380377Z output = model(*input) 2022-11-23T03:54:46.5380708Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5380841Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5381216Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5381389Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5381763Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5381878Z _lazy_init(state, module) 2022-11-23T03:54:46.5382238Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5382374Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5382723Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5382844Z return func(*args, **kwargs) 2022-11-23T03:54:46.5383229Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5383324Z p_assert( 2022-11-23T03:54:46.5383670Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5383787Z traceback.print_stack() 2022-11-23T03:54:46.5383913Z File "", line 1, in 2022-11-23T03:54:46.5384113Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5384245Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5384434Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5384579Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5384779Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5384862Z self.run() 2022-11-23T03:54:46.5385055Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5385194Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5385546Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5385736Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5386107Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5386224Z getattr(self, test_name)() 2022-11-23T03:54:46.5386588Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5386678Z fn() 2022-11-23T03:54:46.5387050Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5387164Z test(self, **param_kwargs) 2022-11-23T03:54:46.5387530Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5387650Z return func(*args, **kwargs) 2022-11-23T03:54:46.5387929Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5388086Z self.run_subtests( 2022-11-23T03:54:46.5388455Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5388609Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5388968Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5389112Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5389498Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5389610Z output = model(*input) 2022-11-23T03:54:46.5389942Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5390076Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5390465Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5390633Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5391012Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5391128Z _lazy_init(state, module) 2022-11-23T03:54:46.5391486Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5391621Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5391962Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5392081Z return func(*args, **kwargs) 2022-11-23T03:54:46.5392466Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5392562Z p_assert( 2022-11-23T03:54:46.5392906Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5393022Z traceback.print_stack() 2022-11-23T03:54:46.5393129Z File "", line 1, in 2022-11-23T03:54:46.5393332Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5393468Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5393665Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5393808Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5394014Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5394113Z self.run() 2022-11-23T03:54:46.5394306Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5394442Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5394788Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5394996Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5395369Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5395487Z getattr(self, test_name)() 2022-11-23T03:54:46.5395853Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5395943Z fn() 2022-11-23T03:54:46.5396316Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5396432Z test(self, **param_kwargs) 2022-11-23T03:54:46.5396782Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5396903Z return func(*args, **kwargs) 2022-11-23T03:54:46.5397233Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5397351Z self.run_subtests( 2022-11-23T03:54:46.5397715Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5397868Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5398239Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5398385Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5398764Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5398877Z output = model(*input) 2022-11-23T03:54:46.5399213Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5399346Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5399733Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5399902Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5400275Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5400390Z _lazy_init(state, module) 2022-11-23T03:54:46.5400746Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5400882Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5401227Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5401332Z return func(*args, **kwargs) 2022-11-23T03:54:46.5401717Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5401809Z p_assert( 2022-11-23T03:54:46.5402153Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5402267Z traceback.print_stack() 2022-11-23T03:54:46.5402387Z File "", line 1, in 2022-11-23T03:54:46.5402585Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5402716Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5402907Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5403047Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5403250Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5403343Z self.run() 2022-11-23T03:54:46.5403533Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5403671Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5404018Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5404214Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5404588Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5404691Z getattr(self, test_name)() 2022-11-23T03:54:46.5405058Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5405150Z fn() 2022-11-23T03:54:46.5405517Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5405635Z test(self, **param_kwargs) 2022-11-23T03:54:46.5405993Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5406110Z return func(*args, **kwargs) 2022-11-23T03:54:46.5406431Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5406539Z self.run_subtests( 2022-11-23T03:54:46.5406907Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5407055Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5407426Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5407564Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5408007Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5408120Z output = model(*input) 2022-11-23T03:54:46.5408448Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5408582Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5408970Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5409124Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5409501Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5409611Z _lazy_init(state, module) 2022-11-23T03:54:46.5409970Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5410104Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5410441Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5410555Z return func(*args, **kwargs) 2022-11-23T03:54:46.5410744Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5410878Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5411226Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5411349Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5411719Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5411829Z getattr(self, test_name)() 2022-11-23T03:54:46.5412197Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5412285Z fn() 2022-11-23T03:54:46.5412658Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5412771Z test(self, **param_kwargs) 2022-11-23T03:54:46.5413133Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5413237Z return func(*args, **kwargs) 2022-11-23T03:54:46.5413577Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5413683Z self.run_subtests( 2022-11-23T03:54:46.5414043Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5414201Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5414572Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5414713Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5415095Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5415203Z output = model(*input) 2022-11-23T03:54:46.5415532Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5415719Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5416105Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5416269Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5416638Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5416749Z _lazy_init(state, module) 2022-11-23T03:54:46.5417106Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5417237Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5417581Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5417697Z return func(*args, **kwargs) 2022-11-23T03:54:46.5418074Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5418167Z p_assert( 2022-11-23T03:54:46.5418504Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5418618Z traceback.print_stack() 2022-11-23T03:54:46.5418737Z File "", line 1, in 2022-11-23T03:54:46.5418933Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5419065Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5419256Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5419398Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5419600Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5419694Z self.run() 2022-11-23T03:54:46.5419883Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5420017Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5420364Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5420486Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5420856Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5420959Z getattr(self, test_name)() 2022-11-23T03:54:46.5421324Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5421414Z fn() 2022-11-23T03:54:46.5421786Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5421903Z test(self, **param_kwargs) 2022-11-23T03:54:46.5422270Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5422389Z return func(*args, **kwargs) 2022-11-23T03:54:46.5422730Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5422839Z self.run_subtests( 2022-11-23T03:54:46.5423204Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5423357Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5423728Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5423870Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5424252Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5424364Z output = model(*input) 2022-11-23T03:54:46.5424698Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5424883Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5425275Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5425441Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5425802Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5425915Z _lazy_init(state, module) 2022-11-23T03:54:46.5426275Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5426411Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5426754Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5426871Z return func(*args, **kwargs) 2022-11-23T03:54:46.5427258Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5427355Z p_assert( 2022-11-23T03:54:46.5427696Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5427813Z traceback.print_stack() 2022-11-23T03:54:46.5427935Z File "", line 1, in 2022-11-23T03:54:46.5428139Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5428273Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5428462Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5428605Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5428809Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5428905Z self.run() 2022-11-23T03:54:46.5429085Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5429226Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5429573Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5429697Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5430065Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5430183Z getattr(self, test_name)() 2022-11-23T03:54:46.5430552Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5430644Z fn() 2022-11-23T03:54:46.5431016Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5431132Z test(self, **param_kwargs) 2022-11-23T03:54:46.5431497Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5431679Z return func(*args, **kwargs) 2022-11-23T03:54:46.5431971Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5432077Z self.run_subtests( 2022-11-23T03:54:46.5432439Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5432594Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5432964Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5433115Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5433485Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5433599Z output = model(*input) 2022-11-23T03:54:46.5433975Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5434114Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5434499Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5434667Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5435041Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5435155Z _lazy_init(state, module) 2022-11-23T03:54:46.5435511Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5435647Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5435988Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5436108Z return func(*args, **kwargs) 2022-11-23T03:54:46.5436891Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.5437665Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.5438435Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.5439207Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.5439970Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.5440732Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.5441498Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.5442342Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.5443102Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.5443903Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.5444670Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.5445442Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:54:46.5445549Z dist init r=0, world=2 2022-11-23T03:54:46.5445852Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.5446160Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.5446464Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.5446766Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.5447069Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.5447372Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.5447684Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.5448033Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.5448337Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.5448638Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.5448940Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.5449297Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.5449695Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5449789Z p_assert( 2022-11-23T03:54:46.5450130Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5450250Z traceback.print_stack() 2022-11-23T03:54:46.5450373Z File "", line 1, in 2022-11-23T03:54:46.5450568Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5450703Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5450894Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5451085Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5451297Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5451394Z self.run() 2022-11-23T03:54:46.5451588Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5451724Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5452077Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5452191Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5452563Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5452680Z getattr(self, test_name)() 2022-11-23T03:54:46.5453046Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5453138Z fn() 2022-11-23T03:54:46.5453515Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5453636Z test(self, **param_kwargs) 2022-11-23T03:54:46.5454003Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5454121Z return func(*args, **kwargs) 2022-11-23T03:54:46.5454399Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5454503Z self.run_subtests( 2022-11-23T03:54:46.5454862Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5455018Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5455393Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5455541Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5455929Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5456041Z output = model(*input) 2022-11-23T03:54:46.5456377Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5456498Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5456881Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5457048Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5457425Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5457540Z _lazy_init(state, module) 2022-11-23T03:54:46.5457898Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5458090Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5458438Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5458553Z return func(*args, **kwargs) 2022-11-23T03:54:46.5458941Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5459033Z p_assert( 2022-11-23T03:54:46.5459376Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5459491Z traceback.print_stack() 2022-11-23T03:54:46.5459592Z dist init r=1, world=2 2022-11-23T03:54:46.5459906Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.5460272Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.5460579Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.5460884Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.5461187Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.5461488Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.5461794Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.5462100Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.5462400Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.5462701Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.5463002Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.5463301Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.5463699Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5463783Z p_assert( 2022-11-23T03:54:46.5464128Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5464246Z traceback.print_stack() 2022-11-23T03:54:46.5464371Z File "", line 1, in 2022-11-23T03:54:46.5464571Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5464706Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5464896Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5465034Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5465235Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5465332Z self.run() 2022-11-23T03:54:46.5465524Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5465717Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5466070Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5466194Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5466560Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5466678Z getattr(self, test_name)() 2022-11-23T03:54:46.5467045Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5467122Z fn() 2022-11-23T03:54:46.5467495Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5467613Z test(self, **param_kwargs) 2022-11-23T03:54:46.5467973Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5468135Z return func(*args, **kwargs) 2022-11-23T03:54:46.5468417Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 128, in test_nested_wrapped_model_single_iteration_mixed_precision 2022-11-23T03:54:46.5468520Z self.run_subtests( 2022-11-23T03:54:46.5468884Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5469035Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5469404Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5469550Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5469932Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5470039Z output = model(*input) 2022-11-23T03:54:46.5470379Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5470512Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5470894Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5471063Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5471431Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5471542Z _lazy_init(state, module) 2022-11-23T03:54:46.5471885Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5472017Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5472359Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5472476Z return func(*args, **kwargs) 2022-11-23T03:54:46.5472868Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5472962Z p_assert( 2022-11-23T03:54:46.5473299Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5473414Z traceback.print_stack() 2022-11-23T03:54:46.5473504Z ok (8.338s) 2022-11-23T03:54:46.5473832Z test_transformer_offload_false_no_shard_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51012 2022-11-23T03:54:46.5474037Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51013 2022-11-23T03:54:46.5474412Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.5474578Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.5474963Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.5475197Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.5475426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.5475800Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.5475965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.5476351Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.5476519Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.5476744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.5477187Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.5477589Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.5477863Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.5478140Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.5478353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.5478566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.5478792Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5479011Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5480067Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.5480167Z warnings.warn( 2022-11-23T03:54:46.5480386Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5481427Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.5481528Z warnings.warn( 2022-11-23T03:54:46.5481743Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5481955Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5482168Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5482381Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5482595Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5482811Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5483022Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5483233Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5483444Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5483707Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5483918Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5484132Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5484333Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5484548Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5484758Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5484970Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5485178Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5485429Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5485646Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5485858Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5486066Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5486277Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5486486Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5486700Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5486911Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5487122Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5487342Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5487555Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5503224Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5503590Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5503815Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5504040Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5504242Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5504464Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5504681Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5504929Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5505151Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5505379Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5505599Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5505823Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5506039Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5506257Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5506473Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5506697Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5507012Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5507125Z dist init r=1, world=2 2022-11-23T03:54:46.5507236Z dist init r=0, world=2 2022-11-23T03:54:46.5507336Z ok (11.441s) 2022-11-23T03:54:46.5507662Z test_transformer_offload_false_none_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51165 2022-11-23T03:54:46.5507878Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51166 2022-11-23T03:54:46.5508467Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.5508644Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.5509024Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.5509271Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.5509509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.5509894Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.5510066Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.5510461Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.5510646Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.5510884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.5511295Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.5511701Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.5512001Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.5512285Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.5512512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.5512734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.5512965Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5513194Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5514257Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.5514379Z warnings.warn( 2022-11-23T03:54:46.5514610Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5515666Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.5515782Z warnings.warn( 2022-11-23T03:54:46.5516011Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5516295Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5516522Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5516748Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5516974Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5517202Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5517402Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5517626Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5517848Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5518122Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5518355Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5518579Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5518802Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5519027Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5519251Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5519478Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5519701Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5519924Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5520154Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5520384Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5520603Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5520823Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5521041Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5521260Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5521477Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5521696Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5521916Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5522122Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5522340Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5522559Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5522779Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5523001Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5523220Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5523442Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5523666Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5523893Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5524166Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5524394Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5524617Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5524839Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5525069Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5525295Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5525515Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5525742Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5526003Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5526127Z dist init r=0, world=2 2022-11-23T03:54:46.5526218Z dist init r=1, world=2 2022-11-23T03:54:46.5526320Z ok (11.741s) 2022-11-23T03:54:46.5526661Z test_transformer_offload_false_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51318 2022-11-23T03:54:46.5526877Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51319 2022-11-23T03:54:46.5527275Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.5527450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.5527895Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.5528085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.5528330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.5528716Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.5528889Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.5529283Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.5529474Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.5529709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.5530126Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.5530529Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.5530821Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.5531109Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.5531333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.5531558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.5531788Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5531989Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5533067Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.5533246Z warnings.warn( 2022-11-23T03:54:46.5533451Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5534532Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.5534642Z warnings.warn( 2022-11-23T03:54:46.5534849Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5535126Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5535366Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5535590Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5535816Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5536037Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5536262Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5536487Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5536711Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5536935Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5537173Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5537398Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5537621Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5537844Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5538067Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5538290Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5538518Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5538745Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5538971Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5539204Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5539433Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5539633Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5539866Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5540092Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5540312Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5540538Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5540760Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5540988Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5541279Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5541504Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5541726Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5541951Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5542173Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5542397Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5542618Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5542842Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5543105Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5543349Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5543578Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5543803Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5544027Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5544228Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5544460Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5544683Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5544908Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5545029Z dist init r=0, world=2 2022-11-23T03:54:46.5545143Z dist init r=1, world=2 2022-11-23T03:54:46.5545245Z ok (11.547s) 2022-11-23T03:54:46.5545576Z test_transformer_offload_true_no_shard_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51471 2022-11-23T03:54:46.5545803Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51472 2022-11-23T03:54:46.5546197Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.5546491Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.5546894Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.5547085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.5547360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.5547755Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.5547933Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.5548332Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.5548519Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.5548756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.5549163Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.5549566Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.5549827Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.5550189Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.5550414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.5550644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.5550871Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5551097Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5552189Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.5552308Z warnings.warn( 2022-11-23T03:54:46.5552440Z File "", line 1, in 2022-11-23T03:54:46.5552653Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5552795Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5553000Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5553154Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5553369Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5553474Z self.run() 2022-11-23T03:54:46.5553680Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5553827Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5554192Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5554335Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5554718Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5554820Z getattr(self, test_name)() 2022-11-23T03:54:46.5555205Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5555307Z fn() 2022-11-23T03:54:46.5555691Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5555816Z test(self, **param_kwargs) 2022-11-23T03:54:46.5556190Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5556318Z return func(*args, **kwargs) 2022-11-23T03:54:46.5556556Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5556674Z self.run_subtests( 2022-11-23T03:54:46.5557048Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5557212Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5557598Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5557753Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5558145Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5558269Z output = model(*input) 2022-11-23T03:54:46.5558621Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5558764Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5559159Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5559376Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5559769Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5559894Z _lazy_init(state, module) 2022-11-23T03:54:46.5560262Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5560408Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5560768Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5560894Z return func(*args, **kwargs) 2022-11-23T03:54:46.5561299Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5561401Z p_assert( 2022-11-23T03:54:46.5561800Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5561934Z traceback.print_stack() 2022-11-23T03:54:46.5562166Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5563221Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.5563336Z warnings.warn( 2022-11-23T03:54:46.5563469Z File "", line 1, in 2022-11-23T03:54:46.5563677Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5563823Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5564032Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5564185Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5564399Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5564481Z self.run() 2022-11-23T03:54:46.5564684Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5564831Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5565191Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5565331Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5565712Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5565838Z getattr(self, test_name)() 2022-11-23T03:54:46.5566217Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5566327Z fn() 2022-11-23T03:54:46.5566720Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5566846Z test(self, **param_kwargs) 2022-11-23T03:54:46.5567221Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5567347Z return func(*args, **kwargs) 2022-11-23T03:54:46.5567587Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5567922Z self.run_subtests( 2022-11-23T03:54:46.5568304Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5568467Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5568854Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5569063Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5569464Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5569588Z output = model(*input) 2022-11-23T03:54:46.5569938Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5570079Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5570477Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5570654Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5571042Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5571168Z _lazy_init(state, module) 2022-11-23T03:54:46.5571592Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5571746Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5572107Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5572237Z return func(*args, **kwargs) 2022-11-23T03:54:46.5572637Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5572743Z p_assert( 2022-11-23T03:54:46.5573096Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5573223Z traceback.print_stack() 2022-11-23T03:54:46.5573455Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5573564Z File "", line 1, in 2022-11-23T03:54:46.5573778Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5573931Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5574132Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5574289Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5574503Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5574608Z self.run() 2022-11-23T03:54:46.5574814Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5574965Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5575325Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5575466Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5575851Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5575976Z getattr(self, test_name)() 2022-11-23T03:54:46.5576363Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5576464Z fn() 2022-11-23T03:54:46.5576855Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5576958Z test(self, **param_kwargs) 2022-11-23T03:54:46.5577335Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5577468Z return func(*args, **kwargs) 2022-11-23T03:54:46.5577706Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5577821Z self.run_subtests( 2022-11-23T03:54:46.5578195Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5578361Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5578814Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5578971Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5579368Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5579492Z output = model(*input) 2022-11-23T03:54:46.5579836Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5579981Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5580377Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5580552Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5580941Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5581115Z _lazy_init(state, module) 2022-11-23T03:54:46.5581498Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5581643Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5581977Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5582108Z return func(*args, **kwargs) 2022-11-23T03:54:46.5582512Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5582619Z p_assert( 2022-11-23T03:54:46.5582972Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5583103Z traceback.print_stack() 2022-11-23T03:54:46.5583332Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5583466Z File "", line 1, in 2022-11-23T03:54:46.5583681Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5583823Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5584033Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5584186Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5584400Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5584507Z self.run() 2022-11-23T03:54:46.5584715Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5584860Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5585222Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5585335Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5585720Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5585852Z getattr(self, test_name)() 2022-11-23T03:54:46.5586234Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5586335Z fn() 2022-11-23T03:54:46.5586721Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5586847Z test(self, **param_kwargs) 2022-11-23T03:54:46.5587227Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5587355Z return func(*args, **kwargs) 2022-11-23T03:54:46.5587597Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5587712Z self.run_subtests( 2022-11-23T03:54:46.5588084Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5588309Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5588697Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5588851Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5589246Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5589369Z output = model(*input) 2022-11-23T03:54:46.5589722Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5589842Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5590236Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5590415Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5590962Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5591096Z _lazy_init(state, module) 2022-11-23T03:54:46.5591473Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5591619Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5591976Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5592105Z return func(*args, **kwargs) 2022-11-23T03:54:46.5592505Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5592610Z p_assert( 2022-11-23T03:54:46.5592961Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5593091Z traceback.print_stack() 2022-11-23T03:54:46.5593324Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5593460Z File "", line 1, in 2022-11-23T03:54:46.5593671Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5593813Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5593992Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5594146Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5594361Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5594468Z self.run() 2022-11-23T03:54:46.5594671Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5594817Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5595178Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5595315Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5595708Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5595834Z getattr(self, test_name)() 2022-11-23T03:54:46.5596210Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5596309Z fn() 2022-11-23T03:54:46.5596693Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5596824Z test(self, **param_kwargs) 2022-11-23T03:54:46.5597206Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5597337Z return func(*args, **kwargs) 2022-11-23T03:54:46.5597575Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5597668Z self.run_subtests( 2022-11-23T03:54:46.5598044Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5598269Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5598652Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5598808Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5599202Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5599329Z output = model(*input) 2022-11-23T03:54:46.5599674Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5599818Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5600215Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5600458Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5600859Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5600983Z _lazy_init(state, module) 2022-11-23T03:54:46.5601355Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5601543Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5601974Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5602124Z return func(*args, **kwargs) 2022-11-23T03:54:46.5602600Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5602725Z p_assert( 2022-11-23T03:54:46.5603120Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5603282Z traceback.print_stack() 2022-11-23T03:54:46.5603555Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5603708Z File "", line 1, in 2022-11-23T03:54:46.5603961Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5604136Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5604381Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5604565Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5604910Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5605066Z self.run() 2022-11-23T03:54:46.5605344Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5605572Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5606219Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5606384Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5606848Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5607006Z getattr(self, test_name)() 2022-11-23T03:54:46.5607431Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5607555Z fn() 2022-11-23T03:54:46.5608063Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5608214Z test(self, **param_kwargs) 2022-11-23T03:54:46.5608674Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5608830Z return func(*args, **kwargs) 2022-11-23T03:54:46.5609117Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5609369Z self.run_subtests( 2022-11-23T03:54:46.5609826Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5610029Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5610495Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5610681Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5611157Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5611303Z output = model(*input) 2022-11-23T03:54:46.5611716Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5611891Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5612432Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5612654Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5613136Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5613258Z _lazy_init(state, module) 2022-11-23T03:54:46.5613704Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5613876Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5614304Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5614454Z return func(*args, **kwargs) 2022-11-23T03:54:46.5614943Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5615076Z p_assert( 2022-11-23T03:54:46.5615501Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5615659Z traceback.print_stack() 2022-11-23T03:54:46.5615948Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5616107Z File "", line 1, in 2022-11-23T03:54:46.5616361Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5616530Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5616773Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5616960Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5617220Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5617342Z self.run() 2022-11-23T03:54:46.5617562Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5617739Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5618180Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5618345Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5618808Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5618962Z getattr(self, test_name)() 2022-11-23T03:54:46.5619422Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5619537Z fn() 2022-11-23T03:54:46.5620002Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5620160Z test(self, **param_kwargs) 2022-11-23T03:54:46.5620610Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5620763Z return func(*args, **kwargs) 2022-11-23T03:54:46.5621129Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5621272Z self.run_subtests( 2022-11-23T03:54:46.5621666Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5621833Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5622215Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5622348Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5622751Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5622875Z output = model(*input) 2022-11-23T03:54:46.5623216Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5623361Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5623812Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5623993Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5624381Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5624505Z _lazy_init(state, module) 2022-11-23T03:54:46.5624877Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5625023Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5625380Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5625510Z return func(*args, **kwargs) 2022-11-23T03:54:46.5625911Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5626024Z p_assert( 2022-11-23T03:54:46.5626380Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5626508Z traceback.print_stack() 2022-11-23T03:54:46.5626739Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5626848Z File "", line 1, in 2022-11-23T03:54:46.5627057Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5627204Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5627408Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5627562Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5627776Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5627882Z self.run() 2022-11-23T03:54:46.5628088Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5628243Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5628599Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5628740Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5629119Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5629251Z getattr(self, test_name)() 2022-11-23T03:54:46.5629633Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5629735Z fn() 2022-11-23T03:54:46.5630120Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5630247Z test(self, **param_kwargs) 2022-11-23T03:54:46.5630599Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5630789Z return func(*args, **kwargs) 2022-11-23T03:54:46.5631026Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5631141Z self.run_subtests( 2022-11-23T03:54:46.5631519Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5631684Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5632072Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5632229Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5632625Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5632750Z output = model(*input) 2022-11-23T03:54:46.5633165Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5633320Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5633721Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5633899Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5634291Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5634414Z _lazy_init(state, module) 2022-11-23T03:54:46.5634785Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5634931Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5635287Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5635391Z return func(*args, **kwargs) 2022-11-23T03:54:46.5635789Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5635898Z p_assert( 2022-11-23T03:54:46.5636251Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5636377Z traceback.print_stack() 2022-11-23T03:54:46.5636607Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5636743Z File "", line 1, in 2022-11-23T03:54:46.5636955Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5637099Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5637303Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5637456Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5637673Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5637790Z self.run() 2022-11-23T03:54:46.5637993Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5638143Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5638503Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5638615Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5638999Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5639127Z getattr(self, test_name)() 2022-11-23T03:54:46.5639502Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5639601Z fn() 2022-11-23T03:54:46.5639990Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5640116Z test(self, **param_kwargs) 2022-11-23T03:54:46.5640568Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5640696Z return func(*args, **kwargs) 2022-11-23T03:54:46.5640934Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5641048Z self.run_subtests( 2022-11-23T03:54:46.5641419Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5641583Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5641965Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5642121Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5642516Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5642666Z output = model(*input) 2022-11-23T03:54:46.5643141Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5643292Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5643777Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5643988Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5644448Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5644596Z _lazy_init(state, module) 2022-11-23T03:54:46.5645043Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5645219Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5645810Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5646207Z return func(*args, **kwargs) 2022-11-23T03:54:46.5646728Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5646856Z p_assert( 2022-11-23T03:54:46.5647283Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5647434Z traceback.print_stack() 2022-11-23T03:54:46.5647756Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5647914Z File "", line 1, in 2022-11-23T03:54:46.5648163Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5648337Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5648585Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5648744Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5649013Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5649142Z self.run() 2022-11-23T03:54:46.5649387Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5649561Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5649996Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5650161Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5650624Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5650771Z getattr(self, test_name)() 2022-11-23T03:54:46.5651230Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5651354Z fn() 2022-11-23T03:54:46.5651825Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5652068Z test(self, **param_kwargs) 2022-11-23T03:54:46.5652524Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5652672Z return func(*args, **kwargs) 2022-11-23T03:54:46.5652955Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5653095Z self.run_subtests( 2022-11-23T03:54:46.5653517Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5653721Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5654186Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5654374Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5654914Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5655075Z output = model(*input) 2022-11-23T03:54:46.5655511Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5655677Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5656153Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5656372Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5656842Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5656993Z _lazy_init(state, module) 2022-11-23T03:54:46.5657440Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5657608Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5658052Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5658207Z return func(*args, **kwargs) 2022-11-23T03:54:46.5658684Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5658807Z p_assert( 2022-11-23T03:54:46.5659205Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5659356Z traceback.print_stack() 2022-11-23T03:54:46.5659633Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5659788Z File "", line 1, in 2022-11-23T03:54:46.5660044Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5660218Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5660462Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5660663Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5660915Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5661042Z self.run() 2022-11-23T03:54:46.5661294Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5661450Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5661809Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5661948Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5662330Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5662458Z getattr(self, test_name)() 2022-11-23T03:54:46.5662837Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5662914Z fn() 2022-11-23T03:54:46.5663367Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5663494Z test(self, **param_kwargs) 2022-11-23T03:54:46.5663871Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5663997Z return func(*args, **kwargs) 2022-11-23T03:54:46.5664234Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5664349Z self.run_subtests( 2022-11-23T03:54:46.5664724Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5664889Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5665275Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5665433Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5665878Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5666007Z output = model(*input) 2022-11-23T03:54:46.5666359Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5666510Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5666912Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5667093Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5667484Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5667584Z _lazy_init(state, module) 2022-11-23T03:54:46.5667953Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5668103Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5668458Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5668586Z return func(*args, **kwargs) 2022-11-23T03:54:46.5668988Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5669094Z p_assert( 2022-11-23T03:54:46.5669446Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5669571Z traceback.print_stack() 2022-11-23T03:54:46.5669800Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5669932Z File "", line 1, in 2022-11-23T03:54:46.5670146Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5670293Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5670501Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5670656Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5670872Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5670977Z self.run() 2022-11-23T03:54:46.5671159Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5671307Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5671667Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5671803Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5672192Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5672318Z getattr(self, test_name)() 2022-11-23T03:54:46.5672702Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5672862Z fn() 2022-11-23T03:54:46.5673250Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5673377Z test(self, **param_kwargs) 2022-11-23T03:54:46.5673755Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5673883Z return func(*args, **kwargs) 2022-11-23T03:54:46.5674122Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5674239Z self.run_subtests( 2022-11-23T03:54:46.5674610Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5674777Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5675209Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5675414Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5675866Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5676014Z output = model(*input) 2022-11-23T03:54:46.5676448Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5676623Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5677106Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5677321Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5677784Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5677930Z _lazy_init(state, module) 2022-11-23T03:54:46.5678390Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5678564Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5679006Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5679155Z return func(*args, **kwargs) 2022-11-23T03:54:46.5679636Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5679768Z p_assert( 2022-11-23T03:54:46.5680195Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5680344Z traceback.print_stack() 2022-11-23T03:54:46.5680619Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5680779Z File "", line 1, in 2022-11-23T03:54:46.5681007Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5681182Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5681428Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5681612Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5681871Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5682001Z self.run() 2022-11-23T03:54:46.5682246Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5682425Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5682855Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5683023Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5683483Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5683635Z getattr(self, test_name)() 2022-11-23T03:54:46.5684193Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5684312Z fn() 2022-11-23T03:54:46.5684772Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5684926Z test(self, **param_kwargs) 2022-11-23T03:54:46.5685378Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5685501Z return func(*args, **kwargs) 2022-11-23T03:54:46.5685793Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5685931Z self.run_subtests( 2022-11-23T03:54:46.5686383Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5686581Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5687118Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5687316Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5687931Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5688081Z output = model(*input) 2022-11-23T03:54:46.5688504Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5688674Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5689150Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5689367Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5689837Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5689999Z _lazy_init(state, module) 2022-11-23T03:54:46.5690443Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5690620Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5691052Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5691175Z return func(*args, **kwargs) 2022-11-23T03:54:46.5691660Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5691798Z p_assert( 2022-11-23T03:54:46.5692225Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5692378Z traceback.print_stack() 2022-11-23T03:54:46.5692656Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5692828Z File "", line 1, in 2022-11-23T03:54:46.5693085Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5693263Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5693510Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5693690Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5693952Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5694083Z self.run() 2022-11-23T03:54:46.5694332Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5694509Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5694936Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5695102Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5695539Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5695779Z getattr(self, test_name)() 2022-11-23T03:54:46.5696236Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5696359Z fn() 2022-11-23T03:54:46.5696827Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5696982Z test(self, **param_kwargs) 2022-11-23T03:54:46.5697438Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5697595Z return func(*args, **kwargs) 2022-11-23T03:54:46.5697881Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5698026Z self.run_subtests( 2022-11-23T03:54:46.5698469Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5698732Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5699209Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5699402Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5699887Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5700033Z output = model(*input) 2022-11-23T03:54:46.5700446Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5700623Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5701071Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5701291Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5701723Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5701850Z _lazy_init(state, module) 2022-11-23T03:54:46.5702221Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5702369Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5702731Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5702859Z return func(*args, **kwargs) 2022-11-23T03:54:46.5703261Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5703369Z p_assert( 2022-11-23T03:54:46.5703722Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5703852Z traceback.print_stack() 2022-11-23T03:54:46.5704093Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5704232Z File "", line 1, in 2022-11-23T03:54:46.5704443Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5704588Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5704796Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5704924Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5705140Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5705245Z self.run() 2022-11-23T03:54:46.5705450Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5705598Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5705963Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5706099Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5706552Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5706680Z getattr(self, test_name)() 2022-11-23T03:54:46.5707059Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5707162Z fn() 2022-11-23T03:54:46.5707548Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5707675Z test(self, **param_kwargs) 2022-11-23T03:54:46.5708053Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5708185Z return func(*args, **kwargs) 2022-11-23T03:54:46.5708423Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5708540Z self.run_subtests( 2022-11-23T03:54:46.5708937Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5709108Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5709493Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5709649Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5710043Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5710165Z output = model(*input) 2022-11-23T03:54:46.5710514Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5710658Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5711059Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5711248Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5711643Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5711770Z _lazy_init(state, module) 2022-11-23T03:54:46.5712141Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5712288Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5712653Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5712782Z return func(*args, **kwargs) 2022-11-23T03:54:46.5713183Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5713289Z p_assert( 2022-11-23T03:54:46.5713646Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5713756Z traceback.print_stack() 2022-11-23T03:54:46.5713987Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5714123Z File "", line 1, in 2022-11-23T03:54:46.5714333Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5714479Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5714684Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5714839Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5715060Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5715166Z self.run() 2022-11-23T03:54:46.5715375Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5715534Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5715899Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5716098Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5716482Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5716610Z getattr(self, test_name)() 2022-11-23T03:54:46.5716989Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5717067Z fn() 2022-11-23T03:54:46.5717456Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5717582Z test(self, **param_kwargs) 2022-11-23T03:54:46.5717959Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5718087Z return func(*args, **kwargs) 2022-11-23T03:54:46.5718376Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5718502Z self.run_subtests( 2022-11-23T03:54:46.5718879Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5719045Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5719430Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5719587Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5719986Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5720109Z output = model(*input) 2022-11-23T03:54:46.5720463Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5720608Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5721012Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5721190Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5721578Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5721703Z _lazy_init(state, module) 2022-11-23T03:54:46.5722051Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5722199Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5722554Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5722685Z return func(*args, **kwargs) 2022-11-23T03:54:46.5723085Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5723194Z p_assert( 2022-11-23T03:54:46.5723559Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5723689Z traceback.print_stack() 2022-11-23T03:54:46.5723923Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5724060Z File "", line 1, in 2022-11-23T03:54:46.5724270Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5724415Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5724620Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5724778Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5724991Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5725102Z self.run() 2022-11-23T03:54:46.5725312Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5725439Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5725861Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5726003Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5726383Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5726513Z getattr(self, test_name)() 2022-11-23T03:54:46.5726892Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5726995Z fn() 2022-11-23T03:54:46.5727381Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5727510Z test(self, **param_kwargs) 2022-11-23T03:54:46.5727933Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5728063Z return func(*args, **kwargs) 2022-11-23T03:54:46.5728367Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5728489Z self.run_subtests( 2022-11-23T03:54:46.5728872Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5729049Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5729436Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5729592Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5729963Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5730088Z output = model(*input) 2022-11-23T03:54:46.5730432Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5730583Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5730979Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5731154Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5731542Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5731670Z _lazy_init(state, module) 2022-11-23T03:54:46.5732038Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5732188Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5732544Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5732672Z return func(*args, **kwargs) 2022-11-23T03:54:46.5733072Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5733185Z p_assert( 2022-11-23T03:54:46.5733537Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5733668Z traceback.print_stack() 2022-11-23T03:54:46.5733897Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5734031Z File "", line 1, in 2022-11-23T03:54:46.5734219Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5734362Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5734566Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5734723Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5734948Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5735055Z self.run() 2022-11-23T03:54:46.5735270Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5735482Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5735848Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5735986Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5736368Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5736497Z getattr(self, test_name)() 2022-11-23T03:54:46.5736879Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5736982Z fn() 2022-11-23T03:54:46.5737366Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5737494Z test(self, **param_kwargs) 2022-11-23T03:54:46.5737916Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5738026Z return func(*args, **kwargs) 2022-11-23T03:54:46.5738266Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5738382Z self.run_subtests( 2022-11-23T03:54:46.5738759Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5738922Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5739305Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5739459Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5739855Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5739978Z output = model(*input) 2022-11-23T03:54:46.5740331Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5740479Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5740884Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5741060Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5741446Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5741572Z _lazy_init(state, module) 2022-11-23T03:54:46.5741948Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5742095Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5742453Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5742581Z return func(*args, **kwargs) 2022-11-23T03:54:46.5742961Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5743072Z p_assert( 2022-11-23T03:54:46.5743433Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5743560Z traceback.print_stack() 2022-11-23T03:54:46.5743795Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5743930Z File "", line 1, in 2022-11-23T03:54:46.5744142Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5744286Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5744490Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5744644Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5744864Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5745035Z self.run() 2022-11-23T03:54:46.5745244Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5745392Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5745759Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5745902Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5746260Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5746394Z getattr(self, test_name)() 2022-11-23T03:54:46.5746772Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5746875Z fn() 2022-11-23T03:54:46.5747262Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5747390Z test(self, **param_kwargs) 2022-11-23T03:54:46.5747821Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5747956Z return func(*args, **kwargs) 2022-11-23T03:54:46.5748195Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5748314Z self.run_subtests( 2022-11-23T03:54:46.5748689Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5748854Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5749240Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5749396Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5749791Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5749924Z output = model(*input) 2022-11-23T03:54:46.5750272Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5750419Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5750792Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5750973Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5751366Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5751490Z _lazy_init(state, module) 2022-11-23T03:54:46.5751863Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5752007Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5752378Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5752516Z return func(*args, **kwargs) 2022-11-23T03:54:46.5752915Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5753023Z p_assert( 2022-11-23T03:54:46.5753380Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5753513Z traceback.print_stack() 2022-11-23T03:54:46.5753748Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5753885Z File "", line 1, in 2022-11-23T03:54:46.5754094Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5754239Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5754445Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5754604Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5754854Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5754967Z self.run() 2022-11-23T03:54:46.5755174Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5755321Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5755687Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5755827Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5756216Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5756346Z getattr(self, test_name)() 2022-11-23T03:54:46.5756730Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5756832Z fn() 2022-11-23T03:54:46.5757281Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5757421Z test(self, **param_kwargs) 2022-11-23T03:54:46.5757808Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5757936Z return func(*args, **kwargs) 2022-11-23T03:54:46.5758184Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5758299Z self.run_subtests( 2022-11-23T03:54:46.5758673Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5758815Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5759204Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5759361Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5759764Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5759890Z output = model(*input) 2022-11-23T03:54:46.5760238Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5760383Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5760784Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5760967Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5761360Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5761489Z _lazy_init(state, module) 2022-11-23T03:54:46.5761860Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5762004Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5762374Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5762505Z return func(*args, **kwargs) 2022-11-23T03:54:46.5762903Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5763009Z p_assert( 2022-11-23T03:54:46.5763362Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5763466Z traceback.print_stack() 2022-11-23T03:54:46.5763706Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5763840Z File "", line 1, in 2022-11-23T03:54:46.5764060Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5764205Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5764417Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5764636Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5764858Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5764967Z self.run() 2022-11-23T03:54:46.5765174Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5765322Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5765686Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5765824Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5766207Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5766337Z getattr(self, test_name)() 2022-11-23T03:54:46.5766715Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5766821Z fn() 2022-11-23T03:54:46.5767228Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5767360Z test(self, **param_kwargs) 2022-11-23T03:54:46.5767876Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5768011Z return func(*args, **kwargs) 2022-11-23T03:54:46.5768257Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5768380Z self.run_subtests( 2022-11-23T03:54:46.5768758Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5768926Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5769315Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5769479Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5769881Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5770006Z output = model(*input) 2022-11-23T03:54:46.5770350Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5770494Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5770892Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5771071Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5771466Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5771593Z _lazy_init(state, module) 2022-11-23T03:54:46.5771943Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5772090Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5772447Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5772580Z return func(*args, **kwargs) 2022-11-23T03:54:46.5772978Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5773084Z p_assert( 2022-11-23T03:54:46.5773439Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5773565Z traceback.print_stack() 2022-11-23T03:54:46.5773797Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5773934Z File "", line 1, in 2022-11-23T03:54:46.5774148Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5774301Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5774590Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5774749Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5774966Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5775077Z self.run() 2022-11-23T03:54:46.5775288Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5775411Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5775787Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5775928Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5776315Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5776444Z getattr(self, test_name)() 2022-11-23T03:54:46.5776878Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5776988Z fn() 2022-11-23T03:54:46.5777381Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5777512Z test(self, **param_kwargs) 2022-11-23T03:54:46.5777887Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5778016Z return func(*args, **kwargs) 2022-11-23T03:54:46.5778255Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5778372Z self.run_subtests( 2022-11-23T03:54:46.5778750Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5778916Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5779305Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5779466Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5779861Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5779959Z output = model(*input) 2022-11-23T03:54:46.5780303Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5780451Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5780853Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5781032Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5781420Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5781554Z _lazy_init(state, module) 2022-11-23T03:54:46.5781933Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5782078Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5782439Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5782572Z return func(*args, **kwargs) 2022-11-23T03:54:46.5782979Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5783084Z p_assert( 2022-11-23T03:54:46.5783435Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5783571Z traceback.print_stack() 2022-11-23T03:54:46.5783809Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5783948Z File "", line 1, in 2022-11-23T03:54:46.5784166Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5784348Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5784556Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5784708Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5784927Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5785033Z self.run() 2022-11-23T03:54:46.5785244Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5785394Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5785760Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5785896Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5786279Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5786461Z getattr(self, test_name)() 2022-11-23T03:54:46.5786854Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5786955Z fn() 2022-11-23T03:54:46.5787343Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5787479Z test(self, **param_kwargs) 2022-11-23T03:54:46.5787861Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5787991Z return func(*args, **kwargs) 2022-11-23T03:54:46.5788207Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5788323Z self.run_subtests( 2022-11-23T03:54:46.5788693Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5788869Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5789263Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5789423Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5789820Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5789945Z output = model(*input) 2022-11-23T03:54:46.5790295Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5790440Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5790832Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5791006Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5791391Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5791515Z _lazy_init(state, module) 2022-11-23T03:54:46.5791882Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5792020Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5792370Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5792492Z return func(*args, **kwargs) 2022-11-23T03:54:46.5792865Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5792966Z p_assert( 2022-11-23T03:54:46.5793321Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5793447Z traceback.print_stack() 2022-11-23T03:54:46.5793685Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5793885Z File "", line 1, in 2022-11-23T03:54:46.5794097Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5794242Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5794450Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5794607Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5794822Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5794931Z self.run() 2022-11-23T03:54:46.5795138Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5795290Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5795652Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5795794Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5796240Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5796350Z getattr(self, test_name)() 2022-11-23T03:54:46.5796740Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5796842Z fn() 2022-11-23T03:54:46.5797225Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5797354Z test(self, **param_kwargs) 2022-11-23T03:54:46.5797730Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5797862Z return func(*args, **kwargs) 2022-11-23T03:54:46.5798102Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5798218Z self.run_subtests( 2022-11-23T03:54:46.5798600Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5798768Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5799163Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5799323Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5799720Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5799847Z output = model(*input) 2022-11-23T03:54:46.5800196Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5800343Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5800743Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5800897Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5801293Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5801419Z _lazy_init(state, module) 2022-11-23T03:54:46.5801794Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5801940Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5802303Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5802435Z return func(*args, **kwargs) 2022-11-23T03:54:46.5802834Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5802941Z p_assert( 2022-11-23T03:54:46.5803297Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5803429Z traceback.print_stack() 2022-11-23T03:54:46.5803730Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5803964Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5804199Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5804433Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5804669Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5804902Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5805135Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5805364Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5805567Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5805844Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5806081Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5806305Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5806530Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5806759Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5806988Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5807215Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5807446Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5807680Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5807967Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5808199Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5808429Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5808658Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5808885Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5809109Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5809337Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5809580Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5809812Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5810042Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5810244Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5810476Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5810700Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5810928Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5811152Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5811379Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5811611Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5811904Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5812133Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5812363Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5812594Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5812825Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5813052Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5813280Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5813510Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5813740Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5814016Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5814252Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5814478Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5814596Z dist init r=1, world=2 2022-11-23T03:54:46.5814900Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.5815223Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.5815551Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.5815881Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.5816204Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.5816524Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.5816841Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.5817157Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.5817476Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.5817792Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.5818113Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.5818426Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.5818544Z dist init r=0, world=2 2022-11-23T03:54:46.5818870Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.5819194Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.5819564Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.5819878Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.5820197Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.5820512Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.5820834Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.5821191Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.5821497Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.5821817Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.5822132Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.5822441Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.5822547Z ok (13.952s) 2022-11-23T03:54:46.5822885Z test_transformer_offload_true_none_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51624 2022-11-23T03:54:46.5823112Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51625 2022-11-23T03:54:46.5823527Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.5823707Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.5824105Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.5824298Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.5824537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.5824930Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.5825110Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.5825511Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.5825704Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.5825939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.5826352Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.5826737Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.5827034Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.5827326Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.5827651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.5827884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.5828118Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5828349Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5829423Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.5829537Z warnings.warn( 2022-11-23T03:54:46.5829728Z File "", line 1, in 2022-11-23T03:54:46.5829951Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5830098Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5830302Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5830462Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5830680Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5830790Z self.run() 2022-11-23T03:54:46.5830999Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5831155Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5831519Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5831654Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5832044Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5832151Z getattr(self, test_name)() 2022-11-23T03:54:46.5832540Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5832642Z fn() 2022-11-23T03:54:46.5833030Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5833158Z test(self, **param_kwargs) 2022-11-23T03:54:46.5833537Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5833672Z return func(*args, **kwargs) 2022-11-23T03:54:46.5833918Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5834035Z self.run_subtests( 2022-11-23T03:54:46.5834417Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5834585Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5834973Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5835135Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5835536Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5835656Z output = model(*input) 2022-11-23T03:54:46.5836003Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5836148Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5836544Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5836696Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5837151Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5837280Z _lazy_init(state, module) 2022-11-23T03:54:46.5837651Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5837801Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5838163Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5838299Z return func(*args, **kwargs) 2022-11-23T03:54:46.5838696Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5838804Z p_assert( 2022-11-23T03:54:46.5839158Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5839297Z traceback.print_stack() 2022-11-23T03:54:46.5839577Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5840655Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.5840772Z warnings.warn( 2022-11-23T03:54:46.5840906Z File "", line 1, in 2022-11-23T03:54:46.5841119Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5841265Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5841469Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5841629Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5841854Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5841938Z self.run() 2022-11-23T03:54:46.5842149Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5842297Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5842657Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5842794Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5843183Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5843313Z getattr(self, test_name)() 2022-11-23T03:54:46.5843701Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5843810Z fn() 2022-11-23T03:54:46.5844200Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5844331Z test(self, **param_kwargs) 2022-11-23T03:54:46.5844714Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5844843Z return func(*args, **kwargs) 2022-11-23T03:54:46.5845087Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5845206Z self.run_subtests( 2022-11-23T03:54:46.5845578Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5845747Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5846106Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5846269Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5846670Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5846855Z output = model(*input) 2022-11-23T03:54:46.5847206Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5847355Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5847794Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5847977Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5848369Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5848499Z _lazy_init(state, module) 2022-11-23T03:54:46.5848874Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5849020Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5849449Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5849589Z return func(*args, **kwargs) 2022-11-23T03:54:46.5850001Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5850107Z p_assert( 2022-11-23T03:54:46.5850461Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5850590Z traceback.print_stack() 2022-11-23T03:54:46.5850799Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5850935Z File "", line 1, in 2022-11-23T03:54:46.5851151Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5851300Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5851509Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5851669Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5851885Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5851995Z self.run() 2022-11-23T03:54:46.5852203Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5852351Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5852712Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5852849Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5853234Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5853363Z getattr(self, test_name)() 2022-11-23T03:54:46.5853743Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5853851Z fn() 2022-11-23T03:54:46.5854241Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5854344Z test(self, **param_kwargs) 2022-11-23T03:54:46.5854724Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5854853Z return func(*args, **kwargs) 2022-11-23T03:54:46.5855094Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5855214Z self.run_subtests( 2022-11-23T03:54:46.5855594Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5855760Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5856145Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5856380Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5856783Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5856907Z output = model(*input) 2022-11-23T03:54:46.5857255Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5857405Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5857806Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5857985Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5858375Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5858504Z _lazy_init(state, module) 2022-11-23T03:54:46.5858926Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5859080Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5859418Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5859546Z return func(*args, **kwargs) 2022-11-23T03:54:46.5859945Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5860052Z p_assert( 2022-11-23T03:54:46.5860405Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5860535Z traceback.print_stack() 2022-11-23T03:54:46.5860767Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5860899Z File "", line 1, in 2022-11-23T03:54:46.5861113Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5861264Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5861478Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5861635Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5861853Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5861962Z self.run() 2022-11-23T03:54:46.5862170Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5862317Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5862654Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5862796Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5863180Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5863310Z getattr(self, test_name)() 2022-11-23T03:54:46.5863696Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5863801Z fn() 2022-11-23T03:54:46.5864187Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5864320Z test(self, **param_kwargs) 2022-11-23T03:54:46.5864699Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5864827Z return func(*args, **kwargs) 2022-11-23T03:54:46.5865075Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5865193Z self.run_subtests( 2022-11-23T03:54:46.5865572Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5865737Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5866127Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5866358Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5866764Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5866892Z output = model(*input) 2022-11-23T03:54:46.5867221Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5867365Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5867769Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5867946Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5868336Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5868462Z _lazy_init(state, module) 2022-11-23T03:54:46.5868887Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5869041Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5869404Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5869537Z return func(*args, **kwargs) 2022-11-23T03:54:46.5869940Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5870050Z p_assert( 2022-11-23T03:54:46.5870405Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5870535Z traceback.print_stack() 2022-11-23T03:54:46.5870767Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5870904Z File "", line 1, in 2022-11-23T03:54:46.5871121Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5871267Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5871446Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5871604Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5871822Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5871931Z self.run() 2022-11-23T03:54:46.5872138Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5872285Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5872647Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5872784Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5873170Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5873307Z getattr(self, test_name)() 2022-11-23T03:54:46.5873689Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5873789Z fn() 2022-11-23T03:54:46.5874177Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5874306Z test(self, **param_kwargs) 2022-11-23T03:54:46.5874684Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5874815Z return func(*args, **kwargs) 2022-11-23T03:54:46.5875055Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5875148Z self.run_subtests( 2022-11-23T03:54:46.5875522Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5875693Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5876155Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5876314Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5876712Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5876837Z output = model(*input) 2022-11-23T03:54:46.5877188Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5877336Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5877740Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5877919Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5878353Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5878487Z _lazy_init(state, module) 2022-11-23T03:54:46.5878871Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5879017Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5879375Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5879511Z return func(*args, **kwargs) 2022-11-23T03:54:46.5879911Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5880023Z p_assert( 2022-11-23T03:54:46.5880350Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5880484Z traceback.print_stack() 2022-11-23T03:54:46.5880716Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5880862Z File "", line 1, in 2022-11-23T03:54:46.5881078Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5881222Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5881427Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5881581Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5881803Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5881910Z self.run() 2022-11-23T03:54:46.5882117Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5882267Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5882631Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5882770Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5883161Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5883294Z getattr(self, test_name)() 2022-11-23T03:54:46.5883650Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5883759Z fn() 2022-11-23T03:54:46.5884144Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5884270Z test(self, **param_kwargs) 2022-11-23T03:54:46.5884656Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5884785Z return func(*args, **kwargs) 2022-11-23T03:54:46.5885023Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5885141Z self.run_subtests( 2022-11-23T03:54:46.5885519Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5885751Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5886140Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5886297Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5886691Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5886814Z output = model(*input) 2022-11-23T03:54:46.5887164Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5887313Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5887855Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5888037Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5888476Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5888610Z _lazy_init(state, module) 2022-11-23T03:54:46.5888989Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5889136Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5889498Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5889632Z return func(*args, **kwargs) 2022-11-23T03:54:46.5890034Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5890140Z p_assert( 2022-11-23T03:54:46.5890496Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5890628Z traceback.print_stack() 2022-11-23T03:54:46.5890877Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5891012Z File "", line 1, in 2022-11-23T03:54:46.5891226Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5891376Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5891584Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5891739Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5891956Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5892038Z self.run() 2022-11-23T03:54:46.5892246Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5892394Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5892760Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5892903Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5893287Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5893414Z getattr(self, test_name)() 2022-11-23T03:54:46.5893794Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5893897Z fn() 2022-11-23T03:54:46.5894287Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5894416Z test(self, **param_kwargs) 2022-11-23T03:54:46.5894795Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5894927Z return func(*args, **kwargs) 2022-11-23T03:54:46.5895170Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5895367Z self.run_subtests( 2022-11-23T03:54:46.5895754Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5895920Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5896306Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5896466Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5896945Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5897093Z output = model(*input) 2022-11-23T03:54:46.5897514Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5897690Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5898195Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5898499Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5898983Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5899129Z _lazy_init(state, module) 2022-11-23T03:54:46.5899577Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5899757Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5900186Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5900346Z return func(*args, **kwargs) 2022-11-23T03:54:46.5900823Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5900947Z p_assert( 2022-11-23T03:54:46.5901383Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5901538Z traceback.print_stack() 2022-11-23T03:54:46.5901816Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5901949Z File "", line 1, in 2022-11-23T03:54:46.5902210Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5902386Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5902637Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5902820Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5903083Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5903223Z self.run() 2022-11-23T03:54:46.5903470Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5903654Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5904096Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5904262Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5904718Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5904868Z getattr(self, test_name)() 2022-11-23T03:54:46.5905329Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5905450Z fn() 2022-11-23T03:54:46.5905915Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5906067Z test(self, **param_kwargs) 2022-11-23T03:54:46.5906492Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5906646Z return func(*args, **kwargs) 2022-11-23T03:54:46.5906936Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5907255Z self.run_subtests( 2022-11-23T03:54:46.5907712Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5907907Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5908371Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5908562Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5909041Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5909189Z output = model(*input) 2022-11-23T03:54:46.5909608Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5909782Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5910316Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5910553Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5911022Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5911169Z _lazy_init(state, module) 2022-11-23T03:54:46.5911619Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5911798Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5912164Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5912293Z return func(*args, **kwargs) 2022-11-23T03:54:46.5912690Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5912801Z p_assert( 2022-11-23T03:54:46.5913164Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5913295Z traceback.print_stack() 2022-11-23T03:54:46.5913530Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5913668Z File "", line 1, in 2022-11-23T03:54:46.5913879Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5914024Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5914227Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5914383Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5914606Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5914718Z self.run() 2022-11-23T03:54:46.5914926Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5915080Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5915444Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5915556Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5915942Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5916071Z getattr(self, test_name)() 2022-11-23T03:54:46.5916453Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5916566Z fn() 2022-11-23T03:54:46.5916952Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5917080Z test(self, **param_kwargs) 2022-11-23T03:54:46.5917456Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5917661Z return func(*args, **kwargs) 2022-11-23T03:54:46.5917902Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5918024Z self.run_subtests( 2022-11-23T03:54:46.5918400Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5918569Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5918953Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5919110Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5919508Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5919656Z output = model(*input) 2022-11-23T03:54:46.5920070Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5920278Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5920769Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5920984Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5921481Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5921636Z _lazy_init(state, module) 2022-11-23T03:54:46.5922080Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5922259Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5922689Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5922839Z return func(*args, **kwargs) 2022-11-23T03:54:46.5923329Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5923459Z p_assert( 2022-11-23T03:54:46.5923890Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5924045Z traceback.print_stack() 2022-11-23T03:54:46.5924328Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5924483Z File "", line 1, in 2022-11-23T03:54:46.5924741Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5924920Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5925136Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5925323Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5925587Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5925714Z self.run() 2022-11-23T03:54:46.5925968Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5926151Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5926593Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5926755Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5927215Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5927372Z getattr(self, test_name)() 2022-11-23T03:54:46.5927903Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5928034Z fn() 2022-11-23T03:54:46.5928507Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5928657Z test(self, **param_kwargs) 2022-11-23T03:54:46.5929120Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5929379Z return func(*args, **kwargs) 2022-11-23T03:54:46.5929667Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5929782Z self.run_subtests( 2022-11-23T03:54:46.5930247Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5930442Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5930907Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5931096Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5931567Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5931715Z output = model(*input) 2022-11-23T03:54:46.5932187Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5932340Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5932744Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5932923Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5933316Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5933441Z _lazy_init(state, module) 2022-11-23T03:54:46.5933811Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5933958Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5934318Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5934455Z return func(*args, **kwargs) 2022-11-23T03:54:46.5934852Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5934960Z p_assert( 2022-11-23T03:54:46.5935286Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5935419Z traceback.print_stack() 2022-11-23T03:54:46.5935650Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5935783Z File "", line 1, in 2022-11-23T03:54:46.5935996Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5936140Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5936351Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5936508Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5936730Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5936846Z self.run() 2022-11-23T03:54:46.5937050Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5937195Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5937559Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5937696Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5938085Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5938217Z getattr(self, test_name)() 2022-11-23T03:54:46.5938599Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5938677Z fn() 2022-11-23T03:54:46.5939068Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5939268Z test(self, **param_kwargs) 2022-11-23T03:54:46.5939659Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5939789Z return func(*args, **kwargs) 2022-11-23T03:54:46.5940033Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5940154Z self.run_subtests( 2022-11-23T03:54:46.5940526Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5940694Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5941082Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5941240Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5941684Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5941817Z output = model(*input) 2022-11-23T03:54:46.5942180Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5942327Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5942723Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5942900Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5943285Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5943385Z _lazy_init(state, module) 2022-11-23T03:54:46.5943761Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5943908Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5944273Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5944409Z return func(*args, **kwargs) 2022-11-23T03:54:46.5944812Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5944920Z p_assert( 2022-11-23T03:54:46.5945275Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5945405Z traceback.print_stack() 2022-11-23T03:54:46.5945637Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5945773Z File "", line 1, in 2022-11-23T03:54:46.5945989Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5946140Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5946344Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5946507Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5946723Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5946833Z self.run() 2022-11-23T03:54:46.5947012Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5947160Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5947522Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5947669Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5948053Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5948182Z getattr(self, test_name)() 2022-11-23T03:54:46.5948564Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5948669Z fn() 2022-11-23T03:54:46.5949061Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5949254Z test(self, **param_kwargs) 2022-11-23T03:54:46.5949634Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5949765Z return func(*args, **kwargs) 2022-11-23T03:54:46.5950011Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5950132Z self.run_subtests( 2022-11-23T03:54:46.5950512Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5950677Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5951063Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5951194Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5951646Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5951779Z output = model(*input) 2022-11-23T03:54:46.5952128Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5952279Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5952679Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5952859Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5953249Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5953375Z _lazy_init(state, module) 2022-11-23T03:54:46.5953756Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5953911Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5954270Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5954402Z return func(*args, **kwargs) 2022-11-23T03:54:46.5954797Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5954906Z p_assert( 2022-11-23T03:54:46.5955261Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5955390Z traceback.print_stack() 2022-11-23T03:54:46.5955622Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5955730Z File "", line 1, in 2022-11-23T03:54:46.5955941Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5956087Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5956302Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5956459Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5956677Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5956788Z self.run() 2022-11-23T03:54:46.5956996Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5957144Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5957507Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5957647Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5958038Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5958170Z getattr(self, test_name)() 2022-11-23T03:54:46.5958554Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5958725Z fn() 2022-11-23T03:54:46.5959115Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5959250Z test(self, **param_kwargs) 2022-11-23T03:54:46.5959608Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5959741Z return func(*args, **kwargs) 2022-11-23T03:54:46.5959985Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5960106Z self.run_subtests( 2022-11-23T03:54:46.5960479Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5960648Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5961036Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5961247Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5961655Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5961783Z output = model(*input) 2022-11-23T03:54:46.5962134Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5962287Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5962684Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5962865Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5963255Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5963380Z _lazy_init(state, module) 2022-11-23T03:54:46.5963757Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5963908Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5964270Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5964373Z return func(*args, **kwargs) 2022-11-23T03:54:46.5964775Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5964881Z p_assert( 2022-11-23T03:54:46.5965235Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5965365Z traceback.print_stack() 2022-11-23T03:54:46.5965608Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5965741Z File "", line 1, in 2022-11-23T03:54:46.5965953Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5966105Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5966314Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5966471Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5966690Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5966799Z self.run() 2022-11-23T03:54:46.5967005Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5967156Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5967516Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5967629Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5968093Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5968223Z getattr(self, test_name)() 2022-11-23T03:54:46.5968685Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5968786Z fn() 2022-11-23T03:54:46.5969172Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5969304Z test(self, **param_kwargs) 2022-11-23T03:54:46.5969680Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5969810Z return func(*args, **kwargs) 2022-11-23T03:54:46.5970050Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5970170Z self.run_subtests( 2022-11-23T03:54:46.5970544Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5970711Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5971162Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5971334Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5971740Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5971867Z output = model(*input) 2022-11-23T03:54:46.5972218Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5972338Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5972742Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5972925Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5973317Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5973450Z _lazy_init(state, module) 2022-11-23T03:54:46.5973824Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5973970Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5974329Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5974456Z return func(*args, **kwargs) 2022-11-23T03:54:46.5974862Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5974976Z p_assert( 2022-11-23T03:54:46.5975331Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5975464Z traceback.print_stack() 2022-11-23T03:54:46.5975694Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5975828Z File "", line 1, in 2022-11-23T03:54:46.5976047Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5976194Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5976402Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5976533Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5976752Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5976865Z self.run() 2022-11-23T03:54:46.5977076Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5977230Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5977599Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5977737Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5978124Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5978337Z getattr(self, test_name)() 2022-11-23T03:54:46.5978723Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5978828Z fn() 2022-11-23T03:54:46.5979215Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5979346Z test(self, **param_kwargs) 2022-11-23T03:54:46.5979722Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5979852Z return func(*args, **kwargs) 2022-11-23T03:54:46.5980093Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5980210Z self.run_subtests( 2022-11-23T03:54:46.5980561Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5980778Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5981174Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5981333Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5981732Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5981858Z output = model(*input) 2022-11-23T03:54:46.5982207Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5982352Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5982754Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5982937Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5983337Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5983468Z _lazy_init(state, module) 2022-11-23T03:54:46.5983840Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5983987Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5984348Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5984478Z return func(*args, **kwargs) 2022-11-23T03:54:46.5984879Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5984986Z p_assert( 2022-11-23T03:54:46.5985342Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5985446Z traceback.print_stack() 2022-11-23T03:54:46.5985686Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5985828Z File "", line 1, in 2022-11-23T03:54:46.5986043Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5986187Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5986391Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5986555Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5986775Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5986883Z self.run() 2022-11-23T03:54:46.5987089Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5987241Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5987603Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5987740Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5988193Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5988325Z getattr(self, test_name)() 2022-11-23T03:54:46.5988706Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5988789Z fn() 2022-11-23T03:54:46.5989173Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5989305Z test(self, **param_kwargs) 2022-11-23T03:54:46.5989684Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5989813Z return func(*args, **kwargs) 2022-11-23T03:54:46.5990054Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.5990170Z self.run_subtests( 2022-11-23T03:54:46.5990596Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.5990772Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.5991161Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.5991325Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.5991722Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.5991850Z output = model(*input) 2022-11-23T03:54:46.5992197Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.5992344Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.5992743Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.5992928Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.5993322Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.5993449Z _lazy_init(state, module) 2022-11-23T03:54:46.5993796Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.5993942Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.5994297Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.5994425Z return func(*args, **kwargs) 2022-11-23T03:54:46.5994833Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.5994943Z p_assert( 2022-11-23T03:54:46.5995297Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.5995429Z traceback.print_stack() 2022-11-23T03:54:46.5995672Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.5995805Z File "", line 1, in 2022-11-23T03:54:46.5996019Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.5996171Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.5996381Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.5996541Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.5996771Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.5996877Z self.run() 2022-11-23T03:54:46.5997057Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.5997205Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.5997569Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.5997776Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.5998167Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.5998298Z getattr(self, test_name)() 2022-11-23T03:54:46.5998681Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.5998783Z fn() 2022-11-23T03:54:46.5999166Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.5999294Z test(self, **param_kwargs) 2022-11-23T03:54:46.5999674Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.5999801Z return func(*args, **kwargs) 2022-11-23T03:54:46.6000041Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.6000211Z self.run_subtests( 2022-11-23T03:54:46.6000607Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.6000775Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.6001158Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.6001319Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.6001690Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.6001816Z output = model(*input) 2022-11-23T03:54:46.6002167Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.6002310Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.6002718Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.6002898Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.6003285Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.6003412Z _lazy_init(state, module) 2022-11-23T03:54:46.6003784Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.6003933Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.6004291Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.6004428Z return func(*args, **kwargs) 2022-11-23T03:54:46.6004827Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.6004938Z p_assert( 2022-11-23T03:54:46.6005297Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.6005430Z traceback.print_stack() 2022-11-23T03:54:46.6005662Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6005799Z File "", line 1, in 2022-11-23T03:54:46.6005985Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.6006129Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.6006351Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.6006508Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.6006725Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.6006833Z self.run() 2022-11-23T03:54:46.6007043Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.6007189Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.6007621Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.6007908Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.6008296Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.6008422Z getattr(self, test_name)() 2022-11-23T03:54:46.6008806Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.6008908Z fn() 2022-11-23T03:54:46.6009293Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.6009426Z test(self, **param_kwargs) 2022-11-23T03:54:46.6009804Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.6009907Z return func(*args, **kwargs) 2022-11-23T03:54:46.6010220Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.6010347Z self.run_subtests( 2022-11-23T03:54:46.6010728Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.6010895Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.6011283Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.6011438Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.6011835Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.6011958Z output = model(*input) 2022-11-23T03:54:46.6012307Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.6012461Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.6012865Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.6013047Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.6013436Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.6013564Z _lazy_init(state, module) 2022-11-23T03:54:46.6013935Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.6014080Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.6014438Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.6014543Z return func(*args, **kwargs) 2022-11-23T03:54:46.6014943Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.6015056Z p_assert( 2022-11-23T03:54:46.6015410Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.6015539Z traceback.print_stack() 2022-11-23T03:54:46.6015771Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6015905Z File "", line 1, in 2022-11-23T03:54:46.6016122Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.6016265Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.6016473Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.6016630Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.6016849Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.6016956Z self.run() 2022-11-23T03:54:46.6017163Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.6017387Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.6017754Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.6017892Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.6018257Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.6018388Z getattr(self, test_name)() 2022-11-23T03:54:46.6018767Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.6018870Z fn() 2022-11-23T03:54:46.6019259Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.6019386Z test(self, **param_kwargs) 2022-11-23T03:54:46.6019764Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.6019973Z return func(*args, **kwargs) 2022-11-23T03:54:46.6020220Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.6020338Z self.run_subtests( 2022-11-23T03:54:46.6020715Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.6020881Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.6021267Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.6021424Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.6021818Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.6021941Z output = model(*input) 2022-11-23T03:54:46.6022292Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.6022440Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.6022815Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.6022998Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.6023390Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.6023514Z _lazy_init(state, module) 2022-11-23T03:54:46.6023887Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.6024039Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.6024401Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.6024531Z return func(*args, **kwargs) 2022-11-23T03:54:46.6024932Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.6025045Z p_assert( 2022-11-23T03:54:46.6025400Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.6025531Z traceback.print_stack() 2022-11-23T03:54:46.6025764Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6025901Z File "", line 1, in 2022-11-23T03:54:46.6026114Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.6026262Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.6026466Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.6026595Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.6026812Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.6026991Z self.run() 2022-11-23T03:54:46.6027202Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.6027356Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.6027726Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.6027864Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.6028255Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.6028384Z getattr(self, test_name)() 2022-11-23T03:54:46.6028766Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.6028870Z fn() 2022-11-23T03:54:46.6029256Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.6029384Z test(self, **param_kwargs) 2022-11-23T03:54:46.6029812Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.6029946Z return func(*args, **kwargs) 2022-11-23T03:54:46.6030197Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.6030314Z self.run_subtests( 2022-11-23T03:54:46.6030666Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.6030832Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.6031218Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.6031373Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.6031774Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.6031898Z output = model(*input) 2022-11-23T03:54:46.6032253Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.6032404Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.6032802Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.6032981Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.6033369Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.6033500Z _lazy_init(state, module) 2022-11-23T03:54:46.6033869Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.6034014Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.6034371Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.6034507Z return func(*args, **kwargs) 2022-11-23T03:54:46.6034905Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.6035014Z p_assert( 2022-11-23T03:54:46.6035366Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.6035470Z traceback.print_stack() 2022-11-23T03:54:46.6035715Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6035848Z File "", line 1, in 2022-11-23T03:54:46.6036060Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.6036209Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.6036416Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.6036570Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.6036877Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.6036986Z self.run() 2022-11-23T03:54:46.6037191Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.6037340Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.6037711Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.6037854Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.6038237Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.6038368Z getattr(self, test_name)() 2022-11-23T03:54:46.6038748Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.6038855Z fn() 2022-11-23T03:54:46.6039215Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.6039404Z test(self, **param_kwargs) 2022-11-23T03:54:46.6039791Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.6039923Z return func(*args, **kwargs) 2022-11-23T03:54:46.6040164Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.6040282Z self.run_subtests( 2022-11-23T03:54:46.6040652Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.6040818Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.6041205Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.6041365Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.6041771Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.6041897Z output = model(*input) 2022-11-23T03:54:46.6042249Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.6042396Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.6042794Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.6042977Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.6043367Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.6043494Z _lazy_init(state, module) 2022-11-23T03:54:46.6043843Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.6043991Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.6044358Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.6044494Z return func(*args, **kwargs) 2022-11-23T03:54:46.6044897Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.6045005Z p_assert( 2022-11-23T03:54:46.6045361Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.6045489Z traceback.print_stack() 2022-11-23T03:54:46.6045720Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6045853Z File "", line 1, in 2022-11-23T03:54:46.6046064Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.6046211Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.6046419Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.6046641Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.6046858Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.6046967Z self.run() 2022-11-23T03:54:46.6047180Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.6047304Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.6047670Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.6047888Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.6048275Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.6048405Z getattr(self, test_name)() 2022-11-23T03:54:46.6048787Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.6048893Z fn() 2022-11-23T03:54:46.6049361Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.6049495Z test(self, **param_kwargs) 2022-11-23T03:54:46.6049877Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.6050012Z return func(*args, **kwargs) 2022-11-23T03:54:46.6050254Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.6050371Z self.run_subtests( 2022-11-23T03:54:46.6050745Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.6050911Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.6051295Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.6051452Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.6051856Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.6051956Z output = model(*input) 2022-11-23T03:54:46.6052303Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.6052447Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.6052850Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.6053030Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.6053428Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.6053553Z _lazy_init(state, module) 2022-11-23T03:54:46.6053925Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.6054076Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.6054433Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.6054562Z return func(*args, **kwargs) 2022-11-23T03:54:46.6054958Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.6055064Z p_assert( 2022-11-23T03:54:46.6055417Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.6055550Z traceback.print_stack() 2022-11-23T03:54:46.6055787Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6055920Z File "", line 1, in 2022-11-23T03:54:46.6056135Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.6056256Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.6056532Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.6056688Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.6056911Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.6057023Z self.run() 2022-11-23T03:54:46.6057229Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.6057382Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.6057746Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.6057885Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.6058269Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.6058397Z getattr(self, test_name)() 2022-11-23T03:54:46.6058824Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.6058934Z fn() 2022-11-23T03:54:46.6059335Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.6059464Z test(self, **param_kwargs) 2022-11-23T03:54:46.6059841Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.6059945Z return func(*args, **kwargs) 2022-11-23T03:54:46.6060185Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.6060301Z self.run_subtests( 2022-11-23T03:54:46.6060675Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.6060839Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.6061227Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.6061390Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.6061789Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.6061912Z output = model(*input) 2022-11-23T03:54:46.6062259Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.6062407Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.6062806Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.6062985Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.6063377Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.6063504Z _lazy_init(state, module) 2022-11-23T03:54:46.6063881Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.6064033Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.6064390Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.6064521Z return func(*args, **kwargs) 2022-11-23T03:54:46.6064900Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.6065011Z p_assert( 2022-11-23T03:54:46.6065367Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.6065499Z traceback.print_stack() 2022-11-23T03:54:46.6065737Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6065870Z File "", line 1, in 2022-11-23T03:54:46.6066087Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.6066297Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.6066502Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.6066658Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.6066876Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.6066983Z self.run() 2022-11-23T03:54:46.6067190Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.6067342Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.6067705Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.6067846Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.6068227Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.6068330Z getattr(self, test_name)() 2022-11-23T03:54:46.6068768Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.6068877Z fn() 2022-11-23T03:54:46.6069269Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.6069400Z test(self, **param_kwargs) 2022-11-23T03:54:46.6069780Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.6069910Z return func(*args, **kwargs) 2022-11-23T03:54:46.6070155Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.6070275Z self.run_subtests( 2022-11-23T03:54:46.6070649Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.6070820Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.6071213Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.6071372Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.6071773Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.6071897Z output = model(*input) 2022-11-23T03:54:46.6072243Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.6072391Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.6072787Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.6072939Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.6073329Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.6073465Z _lazy_init(state, module) 2022-11-23T03:54:46.6073838Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.6073986Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.6074343Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.6074476Z return func(*args, **kwargs) 2022-11-23T03:54:46.6074878Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.6074992Z p_assert( 2022-11-23T03:54:46.6075346Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.6075476Z traceback.print_stack() 2022-11-23T03:54:46.6075709Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6075941Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6076232Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6076464Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6076704Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6076937Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6077169Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6077374Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6077602Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6077830Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6078104Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6078335Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6078562Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6078793Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6079019Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6079245Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6079472Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6079699Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6079927Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6080165Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6080394Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6080621Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6080847Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6081071Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6081304Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6081534Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6081763Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6081994Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6082200Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6082424Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6082650Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6082875Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6083104Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6083331Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6083560Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6083787Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6084019Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6084303Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6084531Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6084757Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6084987Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6085217Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6085445Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6085673Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6085911Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6086182Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6086414Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6086535Z dist init r=0, world=2 2022-11-23T03:54:46.6086838Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.6087160Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.6087478Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.6087934Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.6088263Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.6088582Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.6088894Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.6089207Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.6089519Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.6089835Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.6090161Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.6090470Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.6090586Z dist init r=1, world=2 2022-11-23T03:54:46.6090909Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.6091229Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.6091550Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.6091943Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.6092262Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.6092577Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.6092892Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.6093252Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.6093571Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.6093891Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.6094206Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.6094519Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.6094625Z ok (13.252s) 2022-11-23T03:54:46.6094972Z test_transformer_offload_true_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51777 2022-11-23T03:54:46.6095199Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51778 2022-11-23T03:54:46.6095618Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.6095795Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.6096198Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.6096394Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.6096634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:54:46.6097029Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:54:46.6097209Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:54:46.6097622Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:54:46.6097814Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:54:46.6098054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:54:46.6098442Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.6098859Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:54:46.6099156Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.6099451Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:54:46.6099679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:54:46.6099979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:54:46.6100212Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6100444Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6101526Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.6101646Z warnings.warn( 2022-11-23T03:54:46.6101784Z File "", line 1, in 2022-11-23T03:54:46.6102002Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.6102197Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.6102409Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.6102563Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.6102783Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.6102895Z self.run() 2022-11-23T03:54:46.6103101Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.6103248Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.6103619Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.6103763Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.6104125Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.6104262Z getattr(self, test_name)() 2022-11-23T03:54:46.6104656Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.6104760Z fn() 2022-11-23T03:54:46.6105154Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.6105285Z test(self, **param_kwargs) 2022-11-23T03:54:46.6105667Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.6105800Z return func(*args, **kwargs) 2022-11-23T03:54:46.6106043Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.6106158Z self.run_subtests( 2022-11-23T03:54:46.6106540Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.6106706Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.6107098Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.6107255Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.6107658Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.6107784Z output = model(*input) 2022-11-23T03:54:46.6108136Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.6108286Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.6108661Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.6108843Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.6109237Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.6109430Z _lazy_init(state, module) 2022-11-23T03:54:46.6109813Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.6109962Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.6110336Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.6110469Z return func(*args, **kwargs) 2022-11-23T03:54:46.6110873Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.6110979Z p_assert( 2022-11-23T03:54:46.6111335Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.6111465Z traceback.print_stack() 2022-11-23T03:54:46.6111697Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6112818Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:54:46.6112945Z warnings.warn( 2022-11-23T03:54:46.6113079Z File "", line 1, in 2022-11-23T03:54:46.6113296Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.6113447Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.6113654Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.6113810Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.6114030Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.6114119Z self.run() 2022-11-23T03:54:46.6114332Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.6114481Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.6114851Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.6114992Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.6115378Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.6115509Z getattr(self, test_name)() 2022-11-23T03:54:46.6115900Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.6116005Z fn() 2022-11-23T03:54:46.6116395Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.6116527Z test(self, **param_kwargs) 2022-11-23T03:54:46.6116914Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.6117045Z return func(*args, **kwargs) 2022-11-23T03:54:46.6117284Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.6117404Z self.run_subtests( 2022-11-23T03:54:46.6117782Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.6117949Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.6118311Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.6118469Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.6118868Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.6119074Z output = model(*input) 2022-11-23T03:54:46.6119430Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.6119578Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.6119982Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.6120163Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.6120556Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.6120688Z _lazy_init(state, module) 2022-11-23T03:54:46.6121062Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.6121207Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.6121567Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.6121749Z return func(*args, **kwargs) 2022-11-23T03:54:46.6122167Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.6122273Z p_assert( 2022-11-23T03:54:46.6122629Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.6122764Z traceback.print_stack() 2022-11-23T03:54:46.6122969Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6123106Z File "", line 1, in 2022-11-23T03:54:46.6123318Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.6123464Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.6123669Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.6123829Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.6124054Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.6124163Z self.run() 2022-11-23T03:54:46.6124389Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.6124538Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.6124907Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.6125053Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.6125461Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.6125593Z getattr(self, test_name)() 2022-11-23T03:54:46.6126029Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.6126154Z fn() 2022-11-23T03:54:46.6126636Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.6126791Z test(self, **param_kwargs) 2022-11-23T03:54:46.6127226Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.6127384Z return func(*args, **kwargs) 2022-11-23T03:54:46.6127679Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.6127903Z self.run_subtests( 2022-11-23T03:54:46.6128364Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.6128565Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.6129033Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.6129227Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.6129716Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.6129952Z output = model(*input) 2022-11-23T03:54:46.6130382Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.6130559Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.6131039Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.6131256Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.6131726Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.6131881Z _lazy_init(state, module) 2022-11-23T03:54:46.6132331Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.6132504Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.6133011Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.6133144Z return func(*args, **kwargs) 2022-11-23T03:54:46.6133652Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.6133780Z p_assert( 2022-11-23T03:54:46.6134207Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.6134371Z traceback.print_stack() 2022-11-23T03:54:46.6134656Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6134817Z File "", line 1, in 2022-11-23T03:54:46.6135081Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.6135264Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.6135515Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.6135702Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.6135965Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.6136098Z self.run() 2022-11-23T03:54:46.6136344Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.6136521Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.6136960Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.6137099Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.6137569Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.6137724Z getattr(self, test_name)() 2022-11-23T03:54:46.6138183Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.6138311Z fn() 2022-11-23T03:54:46.6138785Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.6138944Z test(self, **param_kwargs) 2022-11-23T03:54:46.6139402Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.6139558Z return func(*args, **kwargs) 2022-11-23T03:54:46.6139856Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.6139997Z self.run_subtests( 2022-11-23T03:54:46.6140452Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.6140652Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.6141127Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.6141389Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.6141874Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.6142027Z output = model(*input) 2022-11-23T03:54:46.6142393Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.6142513Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.6142912Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.6143095Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.6143488Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.6143625Z _lazy_init(state, module) 2022-11-23T03:54:46.6144052Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.6144210Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.6144578Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.6144711Z return func(*args, **kwargs) 2022-11-23T03:54:46.6145116Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.6145224Z p_assert( 2022-11-23T03:54:46.6145584Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.6145716Z traceback.print_stack() 2022-11-23T03:54:46.6145950Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6146084Z File "", line 1, in 2022-11-23T03:54:46.6146298Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.6146449Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.6146661Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.6146791Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.6147010Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.6147121Z self.run() 2022-11-23T03:54:46.6147328Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.6147477Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.6147840Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.6147987Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.6148373Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.6148502Z getattr(self, test_name)() 2022-11-23T03:54:46.6148891Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.6148997Z fn() 2022-11-23T03:54:46.6149388Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.6149517Z test(self, **param_kwargs) 2022-11-23T03:54:46.6149898Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.6150030Z return func(*args, **kwargs) 2022-11-23T03:54:46.6150275Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.6150395Z self.run_subtests( 2022-11-23T03:54:46.6150744Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.6150911Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.6151306Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.6151526Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.6151928Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.6152053Z output = model(*input) 2022-11-23T03:54:46.6152402Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.6152555Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.6152959Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.6153136Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.6153527Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.6153658Z _lazy_init(state, module) 2022-11-23T03:54:46.6154097Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.6154251Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.6154616Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.6154747Z return func(*args, **kwargs) 2022-11-23T03:54:46.6155154Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.6155265Z p_assert( 2022-11-23T03:54:46.6155594Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.6155728Z traceback.print_stack() 2022-11-23T03:54:46.6155963Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6156101Z File "", line 1, in 2022-11-23T03:54:46.6156317Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.6156471Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.6156683Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.6156840Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.6157057Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.6157170Z self.run() 2022-11-23T03:54:46.6157384Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.6157534Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.6157898Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.6158039Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.6158425Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.6158564Z getattr(self, test_name)() 2022-11-23T03:54:46.6158947Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.6159024Z fn() 2022-11-23T03:54:46.6159417Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.6159545Z test(self, **param_kwargs) 2022-11-23T03:54:46.6159936Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.6160068Z return func(*args, **kwargs) 2022-11-23T03:54:46.6160310Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.6160433Z self.run_subtests( 2022-11-23T03:54:46.6160807Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.6160973Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.6161436Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.6161597Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.6162001Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.6162127Z output = model(*input) 2022-11-23T03:54:46.6162479Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.6162627Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.6163029Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.6163212Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.6163652Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.6163760Z _lazy_init(state, module) 2022-11-23T03:54:46.6164147Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.6164295Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.6164656Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.6164787Z return func(*args, **kwargs) 2022-11-23T03:54:46.6165190Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.6165305Z p_assert( 2022-11-23T03:54:46.6165672Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.6165803Z traceback.print_stack() 2022-11-23T03:54:46.6166041Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6166181Z File "", line 1, in 2022-11-23T03:54:46.6166396Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.6166543Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.6166752Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.6166911Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.6167134Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.6167243Z self.run() 2022-11-23T03:54:46.6167423Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.6167574Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.6168017Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.6168156Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.6168551Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.6168685Z getattr(self, test_name)() 2022-11-23T03:54:46.6169073Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.6169176Z fn() 2022-11-23T03:54:46.6169564Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.6169694Z test(self, **param_kwargs) 2022-11-23T03:54:46.6170076Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.6170209Z return func(*args, **kwargs) 2022-11-23T03:54:46.6170452Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.6170574Z self.run_subtests( 2022-11-23T03:54:46.6170954Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.6171206Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.6171606Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.6171764Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.6172137Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.6172264Z output = model(*input) 2022-11-23T03:54:46.6172618Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.6172762Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.6173162Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.6173345Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.6173798Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.6173931Z _lazy_init(state, module) 2022-11-23T03:54:46.6174309Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.6174456Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.6174823Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.6174953Z return func(*args, **kwargs) 2022-11-23T03:54:46.6175358Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.6175466Z p_assert( 2022-11-23T03:54:46.6175824Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.6175958Z traceback.print_stack() 2022-11-23T03:54:46.6176200Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6176340Z File "", line 1, in 2022-11-23T03:54:46.6176526Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.6176674Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.6176884Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.6177043Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.6177273Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.6177382Z self.run() 2022-11-23T03:54:46.6177590Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.6177741Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.6178109Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.6178255Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.6178647Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.6178782Z getattr(self, test_name)() 2022-11-23T03:54:46.6179165Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.6179269Z fn() 2022-11-23T03:54:46.6179659Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.6179795Z test(self, **param_kwargs) 2022-11-23T03:54:46.6180178Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.6180282Z return func(*args, **kwargs) 2022-11-23T03:54:46.6180527Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.6180649Z self.run_subtests( 2022-11-23T03:54:46.6181100Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.6181268Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.6181659Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.6181817Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.6182216Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.6182342Z output = model(*input) 2022-11-23T03:54:46.6182692Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.6182842Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.6183252Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.6183483Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.6183883Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.6184016Z _lazy_init(state, module) 2022-11-23T03:54:46.6184392Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.6184540Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.6184901Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.6185005Z return func(*args, **kwargs) 2022-11-23T03:54:46.6185405Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.6185515Z p_assert( 2022-11-23T03:54:46.6185873Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.6186014Z traceback.print_stack() 2022-11-23T03:54:46.6186254Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6186393Z File "", line 1, in 2022-11-23T03:54:46.6186608Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.6186756Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.6186959Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.6187115Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.6187331Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.6187441Z self.run() 2022-11-23T03:54:46.6187648Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.6187798Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.6188161Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.6188304Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.6188663Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.6188794Z getattr(self, test_name)() 2022-11-23T03:54:46.6189187Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.6189290Z fn() 2022-11-23T03:54:46.6189678Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.6189809Z test(self, **param_kwargs) 2022-11-23T03:54:46.6190189Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.6190319Z return func(*args, **kwargs) 2022-11-23T03:54:46.6190565Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.6190752Z self.run_subtests( 2022-11-23T03:54:46.6191132Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.6191299Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.6191691Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.6191850Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.6192251Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.6192383Z output = model(*input) 2022-11-23T03:54:46.6192734Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.6192879Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.6193319Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.6193510Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.6193907Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.6194037Z _lazy_init(state, module) 2022-11-23T03:54:46.6194411Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.6194558Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.6194923Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.6195057Z return func(*args, **kwargs) 2022-11-23T03:54:46.6195470Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.6195581Z p_assert( 2022-11-23T03:54:46.6195946Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.6196079Z traceback.print_stack() 2022-11-23T03:54:46.6196314Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6196450Z File "", line 1, in 2022-11-23T03:54:46.6196668Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.6196819Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.6197029Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.6197158Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.6197384Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.6197495Z self.run() 2022-11-23T03:54:46.6197705Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.6197859Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.6198225Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.6198364Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.6198750Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.6198882Z getattr(self, test_name)() 2022-11-23T03:54:46.6199266Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.6199369Z fn() 2022-11-23T03:54:46.6199758Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.6199891Z test(self, **param_kwargs) 2022-11-23T03:54:46.6200271Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.6200474Z return func(*args, **kwargs) 2022-11-23T03:54:46.6200723Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.6200847Z self.run_subtests( 2022-11-23T03:54:46.6201203Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.6201373Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.6201764Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.6201926Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.6202327Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.6202453Z output = model(*input) 2022-11-23T03:54:46.6202806Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.6203007Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.6203417Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.6203597Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.6203992Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.6204121Z _lazy_init(state, module) 2022-11-23T03:54:46.6204498Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.6204646Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.6205007Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.6205141Z return func(*args, **kwargs) 2022-11-23T03:54:46.6205550Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.6205661Z p_assert( 2022-11-23T03:54:46.6206024Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.6206128Z traceback.print_stack() 2022-11-23T03:54:46.6206361Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6206495Z File "", line 1, in 2022-11-23T03:54:46.6206713Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.6206863Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.6207070Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.6207225Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.6207443Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.6207555Z self.run() 2022-11-23T03:54:46.6207906Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.6208057Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.6208425Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.6208565Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.6208952Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.6209086Z getattr(self, test_name)() 2022-11-23T03:54:46.6209473Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.6209551Z fn() 2022-11-23T03:54:46.6209940Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.6210076Z test(self, **param_kwargs) 2022-11-23T03:54:46.6210464Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.6210696Z return func(*args, **kwargs) 2022-11-23T03:54:46.6210938Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.6211056Z self.run_subtests( 2022-11-23T03:54:46.6211439Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.6211606Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.6211995Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.6212162Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.6212565Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.6212706Z output = model(*input) 2022-11-23T03:54:46.6213214Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.6213373Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.6213781Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.6213962Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.6214352Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.6214481Z _lazy_init(state, module) 2022-11-23T03:54:46.6214829Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.6214977Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.6215342Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.6215474Z return func(*args, **kwargs) 2022-11-23T03:54:46.6215889Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.6216000Z p_assert( 2022-11-23T03:54:46.6216357Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.6216487Z traceback.print_stack() 2022-11-23T03:54:46.6216722Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6216857Z File "", line 1, in 2022-11-23T03:54:46.6217071Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.6217221Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.6217426Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.6217581Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.6217805Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.6217921Z self.run() 2022-11-23T03:54:46.6218126Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.6218250Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.6218618Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.6218767Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.6219159Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.6219290Z getattr(self, test_name)() 2022-11-23T03:54:46.6219672Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.6219776Z fn() 2022-11-23T03:54:46.6220165Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.6220370Z test(self, **param_kwargs) 2022-11-23T03:54:46.6220760Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.6220895Z return func(*args, **kwargs) 2022-11-23T03:54:46.6221139Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.6221257Z self.run_subtests( 2022-11-23T03:54:46.6221634Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.6221805Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.6222191Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.6222354Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.6222757Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.6222907Z output = model(*input) 2022-11-23T03:54:46.6223268Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.6223418Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.6223820Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.6224000Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.6224393Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.6224521Z _lazy_init(state, module) 2022-11-23T03:54:46.6224907Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.6225055Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.6225423Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.6225560Z return func(*args, **kwargs) 2022-11-23T03:54:46.6225963Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.6226072Z p_assert( 2022-11-23T03:54:46.6226435Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.6226565Z traceback.print_stack() 2022-11-23T03:54:46.6226801Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6226938Z File "", line 1, in 2022-11-23T03:54:46.6227124Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.6227271Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.6227482Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.6227639Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.6227862Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.6227977Z self.run() 2022-11-23T03:54:46.6228189Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.6228340Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.6228703Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.6228840Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.6229226Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.6229355Z getattr(self, test_name)() 2022-11-23T03:54:46.6229739Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.6229844Z fn() 2022-11-23T03:54:46.6230239Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.6230434Z test(self, **param_kwargs) 2022-11-23T03:54:46.6230828Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.6230933Z return func(*args, **kwargs) 2022-11-23T03:54:46.6231174Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.6231291Z self.run_subtests( 2022-11-23T03:54:46.6231669Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.6231835Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.6232223Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.6232383Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.6232829Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.6232966Z output = model(*input) 2022-11-23T03:54:46.6233322Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.6233468Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.6233874Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.6234054Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.6234449Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.6234576Z _lazy_init(state, module) 2022-11-23T03:54:46.6234951Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.6235100Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.6235472Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.6235603Z return func(*args, **kwargs) 2022-11-23T03:54:46.6235978Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.6236090Z p_assert( 2022-11-23T03:54:46.6236447Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.6236576Z traceback.print_stack() 2022-11-23T03:54:46.6236817Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6236954Z File "", line 1, in 2022-11-23T03:54:46.6237166Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.6237317Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.6237526Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.6237685Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.6237907Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.6238018Z self.run() 2022-11-23T03:54:46.6238228Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.6238382Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.6238750Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.6238893Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.6239251Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.6239384Z getattr(self, test_name)() 2022-11-23T03:54:46.6239772Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.6239944Z fn() 2022-11-23T03:54:46.6240339Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.6240471Z test(self, **param_kwargs) 2022-11-23T03:54:46.6240857Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.6240989Z return func(*args, **kwargs) 2022-11-23T03:54:46.6241238Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.6241359Z self.run_subtests( 2022-11-23T03:54:46.6241734Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.6241908Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.6242304Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.6242516Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.6242926Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.6243054Z output = model(*input) 2022-11-23T03:54:46.6243410Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.6243557Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.6243933Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.6244115Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.6244507Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.6244636Z _lazy_init(state, module) 2022-11-23T03:54:46.6245022Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.6245174Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.6245538Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.6245678Z return func(*args, **kwargs) 2022-11-23T03:54:46.6246082Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.6246189Z p_assert( 2022-11-23T03:54:46.6246551Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.6246684Z traceback.print_stack() 2022-11-23T03:54:46.6246920Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6247059Z File "", line 1, in 2022-11-23T03:54:46.6247274Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.6247430Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.6247634Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.6247856Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.6248048Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.6248159Z self.run() 2022-11-23T03:54:46.6248372Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.6248524Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.6248890Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.6249029Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.6249414Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.6249545Z getattr(self, test_name)() 2022-11-23T03:54:46.6249932Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.6250108Z fn() 2022-11-23T03:54:46.6250501Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.6250631Z test(self, **param_kwargs) 2022-11-23T03:54:46.6251009Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.6251139Z return func(*args, **kwargs) 2022-11-23T03:54:46.6251384Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.6251503Z self.run_subtests( 2022-11-23T03:54:46.6251884Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.6252023Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.6252466Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.6252636Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.6253038Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.6253165Z output = model(*input) 2022-11-23T03:54:46.6253519Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.6253666Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.6254068Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.6254254Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.6254643Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.6254778Z _lazy_init(state, module) 2022-11-23T03:54:46.6255162Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.6255310Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.6255673Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.6255812Z return func(*args, **kwargs) 2022-11-23T03:54:46.6256215Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.6256324Z p_assert( 2022-11-23T03:54:46.6256680Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.6256785Z traceback.print_stack() 2022-11-23T03:54:46.6257020Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6257154Z File "", line 1, in 2022-11-23T03:54:46.6257376Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.6257527Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.6257736Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.6257892Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.6258111Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.6258223Z self.run() 2022-11-23T03:54:46.6258432Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.6258582Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.6258949Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.6259088Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.6259480Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.6259686Z getattr(self, test_name)() 2022-11-23T03:54:46.6260081Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.6260184Z fn() 2022-11-23T03:54:46.6260547Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.6260680Z test(self, **param_kwargs) 2022-11-23T03:54:46.6261057Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.6261190Z return func(*args, **kwargs) 2022-11-23T03:54:46.6261433Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.6261552Z self.run_subtests( 2022-11-23T03:54:46.6261932Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.6262154Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.6262549Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.6262712Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.6263114Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.6263241Z output = model(*input) 2022-11-23T03:54:46.6263592Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.6263738Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.6264134Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.6264315Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.6264713Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.6264843Z _lazy_init(state, module) 2022-11-23T03:54:46.6265220Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.6265340Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.6265699Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.6265830Z return func(*args, **kwargs) 2022-11-23T03:54:46.6266244Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.6266355Z p_assert( 2022-11-23T03:54:46.6266715Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.6266849Z traceback.print_stack() 2022-11-23T03:54:46.6267082Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6267225Z File "", line 1, in 2022-11-23T03:54:46.6267434Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.6267582Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.6267793Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.6267949Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.6268166Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.6268274Z self.run() 2022-11-23T03:54:46.6268486Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.6268609Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.6268968Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.6269106Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.6269495Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.6269709Z getattr(self, test_name)() 2022-11-23T03:54:46.6270096Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.6270199Z fn() 2022-11-23T03:54:46.6270589Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.6270719Z test(self, **param_kwargs) 2022-11-23T03:54:46.6271108Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.6271239Z return func(*args, **kwargs) 2022-11-23T03:54:46.6271477Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.6271598Z self.run_subtests( 2022-11-23T03:54:46.6272027Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.6272207Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.6272600Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.6272760Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.6273161Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.6273260Z output = model(*input) 2022-11-23T03:54:46.6273610Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.6273756Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.6274164Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.6274343Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.6274741Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.6274871Z _lazy_init(state, module) 2022-11-23T03:54:46.6275247Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.6275396Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.6275759Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.6275889Z return func(*args, **kwargs) 2022-11-23T03:54:46.6276293Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.6276404Z p_assert( 2022-11-23T03:54:46.6276764Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.6276896Z traceback.print_stack() 2022-11-23T03:54:46.6277144Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6277281Z File "", line 1, in 2022-11-23T03:54:46.6277493Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.6277614Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.6277819Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.6277977Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.6278193Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.6278302Z self.run() 2022-11-23T03:54:46.6278510Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.6278663Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.6279027Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.6279237Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.6279633Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.6279762Z getattr(self, test_name)() 2022-11-23T03:54:46.6280144Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.6280248Z fn() 2022-11-23T03:54:46.6280637Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.6280770Z test(self, **param_kwargs) 2022-11-23T03:54:46.6281153Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.6281284Z return func(*args, **kwargs) 2022-11-23T03:54:46.6281497Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.6281668Z self.run_subtests( 2022-11-23T03:54:46.6282059Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.6282226Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.6282615Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.6282781Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.6283184Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.6283308Z output = model(*input) 2022-11-23T03:54:46.6283657Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.6283806Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.6284210Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.6284393Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.6284786Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.6284915Z _lazy_init(state, module) 2022-11-23T03:54:46.6285291Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.6285442Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.6285802Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.6285933Z return func(*args, **kwargs) 2022-11-23T03:54:46.6286309Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.6286420Z p_assert( 2022-11-23T03:54:46.6286781Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.6286914Z traceback.print_stack() 2022-11-23T03:54:46.6287150Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6287286Z File "", line 1, in 2022-11-23T03:54:46.6287502Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.6287653Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.6287998Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.6288160Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.6288380Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.6288489Z self.run() 2022-11-23T03:54:46.6288707Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.6288861Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.6289318Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.6289455Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.6289841Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.6289945Z getattr(self, test_name)() 2022-11-23T03:54:46.6290326Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.6290430Z fn() 2022-11-23T03:54:46.6290820Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.6290952Z test(self, **param_kwargs) 2022-11-23T03:54:46.6291342Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.6291474Z return func(*args, **kwargs) 2022-11-23T03:54:46.6291776Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.6291906Z self.run_subtests( 2022-11-23T03:54:46.6292282Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.6292450Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.6292839Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.6293000Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.6293399Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.6293518Z output = model(*input) 2022-11-23T03:54:46.6293861Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.6294011Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.6294417Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.6294571Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.6294966Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.6295093Z _lazy_init(state, module) 2022-11-23T03:54:46.6295468Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.6295617Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.6295977Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.6296110Z return func(*args, **kwargs) 2022-11-23T03:54:46.6296511Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.6296622Z p_assert( 2022-11-23T03:54:46.6296977Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.6297107Z traceback.print_stack() 2022-11-23T03:54:46.6297342Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6297478Z File "", line 1, in 2022-11-23T03:54:46.6297691Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.6297841Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.6298046Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.6298204Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.6298394Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.6298503Z self.run() 2022-11-23T03:54:46.6298717Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.6298937Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.6299311Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.6299450Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.6299839Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.6299968Z getattr(self, test_name)() 2022-11-23T03:54:46.6300351Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.6300453Z fn() 2022-11-23T03:54:46.6300849Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.6300981Z test(self, **param_kwargs) 2022-11-23T03:54:46.6301364Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.6301543Z return func(*args, **kwargs) 2022-11-23T03:54:46.6301788Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.6301907Z self.run_subtests( 2022-11-23T03:54:46.6302289Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.6302459Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.6302819Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.6302976Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.6303371Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.6303501Z output = model(*input) 2022-11-23T03:54:46.6303852Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.6304005Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.6304405Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.6304591Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.6304983Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.6305111Z _lazy_init(state, module) 2022-11-23T03:54:46.6305485Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.6305632Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.6305993Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.6306128Z return func(*args, **kwargs) 2022-11-23T03:54:46.6306538Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.6306654Z p_assert( 2022-11-23T03:54:46.6307012Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.6307142Z traceback.print_stack() 2022-11-23T03:54:46.6307350Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6307485Z File "", line 1, in 2022-11-23T03:54:46.6307701Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.6307848Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.6308057Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.6308216Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.6308438Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.6308534Z self.run() 2022-11-23T03:54:46.6308805Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.6308945Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.6309298Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.6309425Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.6309797Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.6309916Z getattr(self, test_name)() 2022-11-23T03:54:46.6310283Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.6310374Z fn() 2022-11-23T03:54:46.6310753Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.6310856Z test(self, **param_kwargs) 2022-11-23T03:54:46.6311275Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.6311400Z return func(*args, **kwargs) 2022-11-23T03:54:46.6311627Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.6311735Z self.run_subtests( 2022-11-23T03:54:46.6312111Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.6312266Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.6312642Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.6312790Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.6313174Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.6313287Z output = model(*input) 2022-11-23T03:54:46.6313632Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.6313765Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.6314152Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.6314320Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.6314695Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.6314810Z _lazy_init(state, module) 2022-11-23T03:54:46.6315172Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.6315293Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.6315646Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.6315769Z return func(*args, **kwargs) 2022-11-23T03:54:46.6316159Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.6316254Z p_assert( 2022-11-23T03:54:46.6316598Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.6316715Z traceback.print_stack() 2022-11-23T03:54:46.6316938Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6317058Z File "", line 1, in 2022-11-23T03:54:46.6317259Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.6317394Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.6317599Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.6317741Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.6317948Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.6318109Z self.run() 2022-11-23T03:54:46.6318306Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.6318447Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.6318786Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.6318913Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.6319283Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.6319400Z getattr(self, test_name)() 2022-11-23T03:54:46.6319767Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.6319859Z fn() 2022-11-23T03:54:46.6320233Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.6320401Z test(self, **param_kwargs) 2022-11-23T03:54:46.6320776Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.6320896Z return func(*args, **kwargs) 2022-11-23T03:54:46.6321122Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.6321229Z self.run_subtests( 2022-11-23T03:54:46.6321593Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.6321748Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.6322125Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.6322273Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.6322666Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.6322783Z output = model(*input) 2022-11-23T03:54:46.6323103Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.6323250Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.6323639Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.6323805Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.6324180Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.6324296Z _lazy_init(state, module) 2022-11-23T03:54:46.6324655Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.6324790Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.6325145Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.6325274Z return func(*args, **kwargs) 2022-11-23T03:54:46.6325661Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.6325758Z p_assert( 2022-11-23T03:54:46.6326101Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.6326218Z traceback.print_stack() 2022-11-23T03:54:46.6326437Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6326559Z File "", line 1, in 2022-11-23T03:54:46.6326761Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.6326881Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.6327077Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.6327284Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.6327489Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.6327585Z self.run() 2022-11-23T03:54:46.6327863Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.6328000Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.6328354Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.6328484Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.6328876Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.6328992Z getattr(self, test_name)() 2022-11-23T03:54:46.6329359Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.6329450Z fn() 2022-11-23T03:54:46.6329891Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.6330020Z test(self, **param_kwargs) 2022-11-23T03:54:46.6330391Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.6330508Z return func(*args, **kwargs) 2022-11-23T03:54:46.6330719Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.6330827Z self.run_subtests( 2022-11-23T03:54:46.6331186Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.6331341Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.6331713Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.6331863Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.6332251Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.6332361Z output = model(*input) 2022-11-23T03:54:46.6332697Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.6332832Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.6333216Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.6333383Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.6333758Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.6333872Z _lazy_init(state, module) 2022-11-23T03:54:46.6334230Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.6334383Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.6334736Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.6334856Z return func(*args, **kwargs) 2022-11-23T03:54:46.6335246Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.6335327Z p_assert( 2022-11-23T03:54:46.6335673Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.6335797Z traceback.print_stack() 2022-11-23T03:54:46.6336019Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6336142Z File "", line 1, in 2022-11-23T03:54:46.6336339Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:54:46.6336476Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:54:46.6336742Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:54:46.6336884Z return self._bootstrap(parent_sentinel) 2022-11-23T03:54:46.6337093Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:54:46.6337191Z self.run() 2022-11-23T03:54:46.6337385Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:54:46.6337524Z self._target(*self._args, **self._kwargs) 2022-11-23T03:54:46.6337875Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:54:46.6338004Z self.run_test(test_name, pipe) 2022-11-23T03:54:46.6338377Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:54:46.6338480Z getattr(self, test_name)() 2022-11-23T03:54:46.6338846Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:54:46.6339004Z fn() 2022-11-23T03:54:46.6339389Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:54:46.6339511Z test(self, **param_kwargs) 2022-11-23T03:54:46.6339878Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:54:46.6340008Z return func(*args, **kwargs) 2022-11-23T03:54:46.6340237Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_core.py", line 171, in test_transformer 2022-11-23T03:54:46.6340342Z self.run_subtests( 2022-11-23T03:54:46.6340704Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 754, in run_subtests 2022-11-23T03:54:46.6340858Z test_fn(*test_args, **test_kwargs, **subtest_kwargs) 2022-11-23T03:54:46.6341232Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:54:46.6341385Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:54:46.6341771Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:54:46.6341883Z output = model(*input) 2022-11-23T03:54:46.6342218Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:54:46.6342358Z return forward_call(*input, **kwargs) 2022-11-23T03:54:46.6342745Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:54:46.6342914Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:54:46.6343275Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:54:46.6343390Z _lazy_init(state, module) 2022-11-23T03:54:46.6343752Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:54:46.6343893Z handle.init_flat_param_attributes() 2022-11-23T03:54:46.6344242Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:54:46.6344362Z return func(*args, **kwargs) 2022-11-23T03:54:46.6344748Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:54:46.6344846Z p_assert( 2022-11-23T03:54:46.6345189Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:54:46.6345306Z traceback.print_stack() 2022-11-23T03:54:46.6345533Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6345752Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6345974Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6346256Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6346472Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6346695Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6346913Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6347134Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6347338Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6347559Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6347777Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6348040Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6348260Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6348483Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6348697Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6348914Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6349129Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6349348Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6349562Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6349779Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6350001Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6350231Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6350451Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6350666Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6350882Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6351106Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6351321Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6351539Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6351759Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6351964Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6352182Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6352398Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6352614Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6352829Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6353048Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6353267Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6353482Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6353701Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6353969Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6354189Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6354404Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6354631Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6354847Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6355064Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6355283Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6355499Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6355757Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:54:46.6355870Z dist init r=0, world=2 2022-11-23T03:54:46.6356186Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.6356494Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.6356802Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.6357092Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.6357397Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.6357702Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.6358004Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.6358316Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.6358639Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.6358948Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.6359262Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.6359566Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:54:46.6359672Z dist init r=1, world=2 2022-11-23T03:54:46.6359986Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.6360293Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.6360599Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.6360951Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.6361252Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.6361556Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.6361858Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.6362171Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.6362505Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.6362812Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.6363113Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.6363413Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:54:46.6363508Z ok (13.552s) 2022-11-23T03:54:46.6363527Z 2022-11-23T03:54:46.6363833Z ---------------------------------------------------------------------- 2022-11-23T03:54:46.6363943Z Ran 59 tests in 843.751s 2022-11-23T03:54:46.6363950Z 2022-11-23T03:54:46.6364050Z OK (skipped=5) 2022-11-23T03:54:46.6364060Z 2022-11-23T03:54:46.6364182Z Generating XML reports... 2022-11-23T03:54:46.6364594Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestHooks-20221123034039.xml 2022-11-23T03:54:46.6364999Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestNoGrad-20221123034039.xml 2022-11-23T03:54:46.6365412Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestParamInit-20221123034039.xml 2022-11-23T03:54:46.6365846Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestParityWithDDP-20221123034039.xml 2022-11-23T03:54:46.6365853Z 2022-11-23T03:54:46.6366396Z ##[endgroup] 2022-11-23T03:54:46.6366862Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_core (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_core_90guj51q) 2022-11-23T03:54:46.6366869Z 2022-11-23T03:54:46.6367132Z Running distributed/fsdp/test_fsdp_comm ... [2022-11-23 03:54:45.934912] 2022-11-23T03:54:46.6367619Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:54:45.935291] 2022-11-23T03:55:46.7453299Z 2022-11-23T03:55:46.7454551Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_comm 2022-11-23T03:55:46.7458495Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_comm_uu66hdgq) 2022-11-23T03:55:46.7460821Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_comm 2022-11-23T03:55:46.7461605Z 2022-11-23T03:55:46.7461888Z Running tests... 2022-11-23T03:55:46.7463076Z ---------------------------------------------------------------------- 2022-11-23T03:55:46.7464437Z test_communication_nested_model_False_use_no_sync_False_sharding_strategy_None (__main__.TestCommunication) 2022-11-23T03:55:46.7466593Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51997 2022-11-23T03:55:46.7468878Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51998 2022-11-23T03:55:46.7471370Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:55:46.7473011Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:55:46.7474881Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:55:46.7476205Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:55:46.7477447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:55:46.7479293Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:55:46.7480898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:55:46.7482618Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:55:46.7483936Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:55:46.7485197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:55:46.7487105Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:55:46.7489398Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:55:46.7491447Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:55:46.7493171Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:55:46.7494458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:55:46.7495829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:55:46.7496837Z dist init r=1, world=2 2022-11-23T03:55:46.7497533Z dist init r=0, world=2 2022-11-23T03:55:46.7498414Z ok (7.535s) 2022-11-23T03:55:46.7500018Z test_communication_nested_model_False_use_no_sync_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication) 2022-11-23T03:55:46.7502895Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52150 2022-11-23T03:55:46.7504868Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52151 2022-11-23T03:55:46.7507336Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:55:46.7508992Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:55:46.7511266Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:55:46.7512938Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:55:46.7514404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:55:46.7516264Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:55:46.7517535Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:55:46.7519221Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:55:46.7520546Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:55:46.7521785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:55:46.7523694Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:55:46.7525992Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:55:46.7527631Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:55:46.7529286Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:55:46.7530557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:55:46.7531893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:55:46.7532864Z dist init r=1, world=2 2022-11-23T03:55:46.7533560Z dist init r=0, world=2 2022-11-23T03:55:46.7560586Z ok (6.936s) 2022-11-23T03:55:46.7561131Z test_communication_nested_model_False_use_no_sync_True_sharding_strategy_None (__main__.TestCommunication) 2022-11-23T03:55:46.7562224Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52303 2022-11-23T03:55:46.7562927Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52304 2022-11-23T03:55:46.7563721Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:55:46.7564293Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:55:46.7565030Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:55:46.7565592Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:55:46.7566151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:55:46.7566952Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:55:46.7567536Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:55:46.7568364Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:55:46.7568957Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:55:46.7569523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:55:46.7570319Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:55:46.7571188Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:55:46.7571914Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:55:46.7572547Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:55:46.7573132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:55:46.7573728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:55:46.7574191Z dist init r=1, world=2 2022-11-23T03:55:46.7574495Z dist init r=0, world=2 2022-11-23T03:55:46.7574820Z ok (6.836s) 2022-11-23T03:55:46.7575368Z test_communication_nested_model_False_use_no_sync_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication) 2022-11-23T03:55:46.7576309Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52456 2022-11-23T03:55:46.7576978Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52457 2022-11-23T03:55:46.7577754Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:55:46.7578235Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:55:46.7578927Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:55:46.7579430Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:55:46.7579896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:55:46.7580581Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:55:46.7581057Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:55:46.7581675Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:55:46.7582165Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:55:46.7582616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:55:46.7583385Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:55:46.7584116Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:55:46.7584724Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:55:46.7585255Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:55:46.7585728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:55:46.7586231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:55:46.7586618Z dist init r=0, world=2 2022-11-23T03:55:46.7586870Z dist init r=1, world=2 2022-11-23T03:55:46.7587144Z ok (6.835s) 2022-11-23T03:55:46.7587544Z test_communication_nested_model_True_use_no_sync_False_sharding_strategy_None (__main__.TestCommunication) 2022-11-23T03:55:46.7588288Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52609 2022-11-23T03:55:46.7588842Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52610 2022-11-23T03:55:46.7589483Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:55:46.7589926Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:55:46.7590548Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:55:46.7591046Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:55:46.7591518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:55:46.7592177Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:55:46.7592657Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:55:46.7593279Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:55:46.7593774Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:55:46.7594218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:55:46.7594896Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:55:46.7595619Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:55:46.7596216Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:55:46.7596746Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:55:46.7597217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:55:46.7597787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:55:46.7598143Z dist init r=1, world=2 2022-11-23T03:55:46.7599402Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:55:46.7600215Z warnings.warn( 2022-11-23T03:55:46.7600495Z dist init r=0, world=2 2022-11-23T03:55:46.7601794Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:55:46.7602601Z warnings.warn( 2022-11-23T03:55:46.7602870Z ok (7.737s) 2022-11-23T03:55:46.7603324Z test_communication_nested_model_True_use_no_sync_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication) 2022-11-23T03:55:46.7604117Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52762 2022-11-23T03:55:46.7604679Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52763 2022-11-23T03:55:46.7605305Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:55:46.7626812Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:55:46.7627627Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:55:46.7628169Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:55:46.7628678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:55:46.7629426Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:55:46.7629941Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:55:46.7630626Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:55:46.7631163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:55:46.7631668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:55:46.7632457Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:55:46.7633271Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:55:46.7633929Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:55:46.7634514Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:55:46.7635035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:55:46.7635579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:55:46.7635983Z dist init r=1, world=2 2022-11-23T03:55:46.7637856Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:55:46.7639679Z warnings.warn( 2022-11-23T03:55:46.7640127Z dist init r=0, world=2 2022-11-23T03:55:46.7642502Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:55:46.7643952Z warnings.warn( 2022-11-23T03:55:46.7644380Z ok (7.136s) 2022-11-23T03:55:46.7645207Z test_communication_nested_model_True_use_no_sync_True_sharding_strategy_None (__main__.TestCommunication) 2022-11-23T03:55:46.7646597Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52915 2022-11-23T03:55:46.7647580Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52916 2022-11-23T03:55:46.7649137Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:55:46.7649954Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:55:46.7651059Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:55:46.7651913Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:55:46.7652728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:55:46.7653938Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:55:46.7654760Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:55:46.7655868Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:55:46.7656726Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:55:46.7657546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:55:46.7658867Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:55:46.7660238Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:55:46.7661362Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:55:46.7662361Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:55:46.7663232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:55:46.7664135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:55:46.7664821Z dist init r=0, world=2 2022-11-23T03:55:46.7667281Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:55:46.7668800Z warnings.warn( 2022-11-23T03:55:46.7669277Z dist init r=1, world=2 2022-11-23T03:55:46.7671719Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:55:46.7673390Z warnings.warn( 2022-11-23T03:55:46.7673842Z ok (6.735s) 2022-11-23T03:55:46.7674645Z test_communication_nested_model_True_use_no_sync_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication) 2022-11-23T03:55:46.7676151Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53068 2022-11-23T03:55:46.7677182Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53069 2022-11-23T03:55:46.7678497Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:55:46.7679377Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:55:46.7680542Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:55:46.7681450Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:55:46.7682309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:55:46.7683553Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:55:46.7684412Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:55:46.7685565Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:55:46.7686471Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:55:46.7687335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:55:46.7688802Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:55:46.7690243Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:55:46.7691415Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:55:46.7692448Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:55:46.7693341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:55:46.7694287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:55:46.7695005Z dist init r=0, world=2 2022-11-23T03:55:46.7697579Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:55:46.7699078Z warnings.warn( 2022-11-23T03:55:46.7699551Z dist init r=1, world=2 2022-11-23T03:55:46.7701985Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:55:46.7703504Z warnings.warn( 2022-11-23T03:55:46.7704173Z ok (6.637s) 2022-11-23T03:55:46.7704473Z 2022-11-23T03:55:46.7705058Z ---------------------------------------------------------------------- 2022-11-23T03:55:46.7705739Z Ran 8 tests in 56.390s 2022-11-23T03:55:46.7706066Z 2022-11-23T03:55:46.7706259Z OK 2022-11-23T03:55:46.7706527Z 2022-11-23T03:55:46.7706776Z Generating XML reports... 2022-11-23T03:55:46.7707969Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_comm/TEST-TestCommunication-20221123035447.xml 2022-11-23T03:55:46.7708650Z 2022-11-23T03:55:46.7709369Z ##[endgroup] 2022-11-23T03:55:46.7710606Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_comm_uu66hdgq) 2022-11-23T03:55:46.7711290Z 2022-11-23T03:55:46.7711877Z Running distributed/fsdp/test_fsdp_checkpoint ... [2022-11-23 03:55:46.745873] 2022-11-23T03:55:46.7713640Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:55:46.746552] 2022-11-23T03:57:40.7316045Z 2022-11-23T03:57:40.7319033Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_checkpoint 2022-11-23T03:57:40.7324242Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_checkpoint (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_checkpoint_ix96tacd) 2022-11-23T03:57:40.7336807Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint 2022-11-23T03:57:40.7338082Z 2022-11-23T03:57:40.7338403Z Running tests... 2022-11-23T03:57:40.7339898Z ---------------------------------------------------------------------- 2022-11-23T03:57:40.7341676Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53288 2022-11-23T03:57:40.7343592Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53289 2022-11-23T03:57:40.7345367Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:57:40.7346618Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:57:40.7348400Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:57:40.7401217Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:57:40.7402412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:57:40.7404307Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:57:40.7406093Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:57:40.7407966Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:57:40.7409644Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:57:40.7410872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:57:40.7412730Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:57:40.7414638Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:57:40.7416229Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:57:40.7417671Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:57:40.7418963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:57:40.7420280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:57:40.7421776Z dist init r=1, world=2 2022-11-23T03:57:40.7422504Z dist init r=0, world=2 2022-11-23T03:57:40.7423198Z ok (7.824s) 2022-11-23T03:57:40.7424799Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53441 2022-11-23T03:57:40.7426655Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53442 2022-11-23T03:57:40.7428442Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:57:40.7429712Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:57:40.7431331Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:57:40.7432648Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:57:40.7434121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:57:40.7435939Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:57:40.7437186Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:57:40.7438830Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:57:40.7440129Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:57:40.7441316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:57:40.7443171Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:57:40.7445136Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:57:40.7446768Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:57:40.7448577Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:57:40.7449848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:57:40.7451165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:57:40.7452165Z dist init r=1, world=2 2022-11-23T03:57:40.7452835Z dist init r=0, world=2 2022-11-23T03:57:40.7453512Z ok (6.936s) 2022-11-23T03:57:40.7455095Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53594 2022-11-23T03:57:40.7456905Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53595 2022-11-23T03:57:40.7458177Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:57:40.7458993Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:57:40.7460095Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:57:40.7460939Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:57:40.7461745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:57:40.7462920Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:57:40.7463730Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:57:40.7464826Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:57:40.7465685Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:57:40.7466636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:57:40.7467866Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:57:40.7469164Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:57:40.7470232Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:57:40.7471167Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:57:40.7471990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:57:40.7472852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:57:40.7473502Z dist init r=1, world=2 2022-11-23T03:57:40.7473943Z dist init r=0, world=2 2022-11-23T03:57:40.7474489Z ok (6.642s) 2022-11-23T03:57:40.7475537Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53747 2022-11-23T03:57:40.7476734Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53748 2022-11-23T03:57:40.7477912Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:57:40.7478736Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:57:40.7479835Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:57:40.7480676Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:57:40.7481495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:57:40.7482697Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:57:40.7483510Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:57:40.7484612Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:57:40.7485473Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:57:40.7486296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:57:40.7487529Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:57:40.7488901Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:57:40.7489968Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:57:40.7490916Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:57:40.7491729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:57:40.7492585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:57:40.7493225Z dist init r=1, world=2 2022-11-23T03:57:40.7493678Z dist init r=0, world=2 2022-11-23T03:57:40.7494098Z ok (6.733s) 2022-11-23T03:57:40.7495134Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53900 2022-11-23T03:57:40.7496338Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53901 2022-11-23T03:57:40.7497494Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:57:40.7498452Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:57:40.7499552Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:57:40.7500417Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:57:40.7501221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:57:40.7502401Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:57:40.7503217Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:57:40.7504322Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:57:40.7505180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:57:40.7506098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:57:40.7507354Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:57:40.7508639Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:57:40.7509705Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:57:40.7510636Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:57:40.7511455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:57:40.7512315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:57:40.7512973Z dist init r=0, world=2 2022-11-23T03:57:40.7513431Z dist init r=1, world=2 2022-11-23T03:57:40.7513852Z ok (6.636s) 2022-11-23T03:57:40.7514884Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54053 2022-11-23T03:57:40.7516095Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54054 2022-11-23T03:57:40.7517269Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:57:40.7518095Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:57:40.7519191Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:57:40.7520043Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:57:40.7520859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:57:40.7522044Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:57:40.7522870Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:57:40.7523962Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:57:40.7524814Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:57:40.7525633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:57:40.7526862Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:57:40.7528235Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:57:40.7529429Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:57:40.7530376Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:57:40.7531303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:57:40.7532143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:57:40.7532774Z dist init r=1, world=2 2022-11-23T03:57:40.7533222Z dist init r=0, world=2 2022-11-23T03:57:40.7533652Z ok (6.735s) 2022-11-23T03:57:40.7534644Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54206 2022-11-23T03:57:40.7535824Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54207 2022-11-23T03:57:40.7536968Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:57:40.7537772Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:57:40.7538948Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:57:40.7539799Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:57:40.7540597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:57:40.7541747Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:57:40.7542551Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:57:40.7543619Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:57:40.7544454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:57:40.7545253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:57:40.7546458Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:57:40.7547723Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:57:40.7548764Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:57:40.7549663Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:57:40.7550467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:57:40.7551303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:57:40.7551937Z dist init r=1, world=2 2022-11-23T03:57:40.7552382Z dist init r=0, world=2 2022-11-23T03:57:40.7552811Z ok (6.733s) 2022-11-23T03:57:40.7553815Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54359 2022-11-23T03:57:40.7554997Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54360 2022-11-23T03:57:40.7556135Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:57:40.7556945Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:57:40.7558022Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:57:40.7558863Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:57:40.7559666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:57:40.7560819Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:57:40.7561615Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:57:40.7562796Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:57:40.7563638Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:57:40.7564429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:57:40.7565626Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:57:40.7566883Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:57:40.7568123Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:57:40.7569050Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:57:40.7569841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:57:40.7570823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:57:40.7571459Z dist init r=0, world=2 2022-11-23T03:57:40.7571905Z dist init r=1, world=2 2022-11-23T03:57:40.7572328Z ok (6.737s) 2022-11-23T03:57:40.7573335Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54512 2022-11-23T03:57:40.7574519Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54513 2022-11-23T03:57:40.7575686Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:57:40.7576512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:57:40.7577602Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:57:40.7578186Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:57:40.7578627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:57:40.7579250Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:57:40.7579686Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:57:40.7580251Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:57:40.7580698Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:57:40.7581125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:57:40.7581767Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:57:40.7582454Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:57:40.7583018Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:57:40.7583510Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:57:40.7583931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:57:40.7584389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:57:40.7584734Z dist init r=0, world=2 2022-11-23T03:57:40.7584985Z dist init r=1, world=2 2022-11-23T03:57:40.7585229Z ok (6.733s) 2022-11-23T03:57:40.7585782Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54665 2022-11-23T03:57:40.7586486Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54666 2022-11-23T03:57:40.7587089Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:57:40.7587527Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:57:40.7588103Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:57:40.7588559Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:57:40.7588992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:57:40.7589615Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:57:40.7590051Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:57:40.7590664Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:57:40.7591124Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:57:40.7591559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:57:40.7592209Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:57:40.7592883Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:57:40.7593437Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:57:40.7593931Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:57:40.7594367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:57:40.7594812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:57:40.7595170Z dist init r=0, world=2 2022-11-23T03:57:40.7595415Z dist init r=1, world=2 2022-11-23T03:57:40.7595650Z ok (6.836s) 2022-11-23T03:57:40.7596202Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54818 2022-11-23T03:57:40.7596846Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54819 2022-11-23T03:57:40.7597460Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:57:40.7597886Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:57:40.7598471Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:57:40.7598925Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:57:40.7599374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:57:40.7599996Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:57:40.7600435Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:57:40.7601011Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:57:40.7601457Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:57:40.7601892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:57:40.7602535Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:57:40.7603221Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:57:40.7603836Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:57:40.7604328Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:57:40.7604761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:57:40.7605217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:57:40.7605549Z dist init r=0, world=2 2022-11-23T03:57:40.7605798Z dist init r=1, world=2 2022-11-23T03:57:40.7606032Z ok (6.837s) 2022-11-23T03:57:40.7606573Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54971 2022-11-23T03:57:40.7607212Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54972 2022-11-23T03:57:40.7607937Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:57:40.7608385Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:57:40.7609073Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:57:40.7609614Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:57:40.7610131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:57:40.7610881Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:57:40.7611402Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:57:40.7612094Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:57:40.7612643Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:57:40.7613152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:57:40.7613920Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:57:40.7614733Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:57:40.7615414Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:57:40.7616010Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:57:40.7616527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:57:40.7617076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:57:40.7617478Z dist init r=1, world=2 2022-11-23T03:57:40.7617784Z dist init r=0, world=2 2022-11-23T03:57:40.7618030Z ok (6.736s) 2022-11-23T03:57:40.7618577Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55124 2022-11-23T03:57:40.7619212Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55125 2022-11-23T03:57:40.7619823Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:57:40.7620375Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:57:40.7620954Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:57:40.7621412Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:57:40.7621835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:57:40.7622556Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:57:40.7622992Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:57:40.7623571Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:57:40.7624023Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:57:40.7624456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:57:40.7625099Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:57:40.7625771Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:57:40.7626315Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:57:40.7626866Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:57:40.7627304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:57:40.7627763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:57:40.7628109Z dist init r=1, world=2 2022-11-23T03:57:40.7628355Z dist init r=0, world=2 2022-11-23T03:57:40.7628578Z ok (6.738s) 2022-11-23T03:57:40.7629127Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55277 2022-11-23T03:57:40.7629762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55278 2022-11-23T03:57:40.7630375Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:57:40.7630816Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:57:40.7631397Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:57:40.7631848Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:57:40.7632284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:57:40.7632896Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:57:40.7633332Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:57:40.7633908Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:57:40.7634357Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:57:40.7634797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:57:40.7635446Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:57:40.7636112Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:57:40.7636670Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:57:40.7637152Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:57:40.7637589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:57:40.7638050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:57:40.7638392Z dist init r=1, world=2 2022-11-23T03:57:40.7638642Z dist init r=0, world=2 2022-11-23T03:57:40.7638880Z ok (7.441s) 2022-11-23T03:57:40.7639414Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55430 2022-11-23T03:57:40.7640116Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55431 2022-11-23T03:57:40.7640731Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:57:40.7641167Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:57:40.7641739Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:57:40.7642197Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:57:40.7642631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:57:40.7643309Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:57:40.7643736Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:57:40.7644312Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:57:40.7644769Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:57:40.7645200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:57:40.7645845Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:57:40.7646514Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:57:40.7647074Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:57:40.7647560Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:57:40.7648046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:57:40.7648529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:57:40.7648930Z dist init r=0, world=2 2022-11-23T03:57:40.7649234Z dist init r=1, world=2 2022-11-23T03:57:40.7649512Z ok (6.737s) 2022-11-23T03:57:40.7650169Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55583 2022-11-23T03:57:40.7650918Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55584 2022-11-23T03:57:40.7651654Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:57:40.7652184Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:57:40.7652883Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:57:40.7653436Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:57:40.7653956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:57:40.7654704Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:57:40.7655217Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:57:40.7655908Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:57:40.7656454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:57:40.7656973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:57:40.7657766Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:57:40.7658533Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:57:40.7659088Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:57:40.7659585Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:57:40.7660004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:57:40.7660459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:57:40.7660804Z dist init r=1, world=2 2022-11-23T03:57:40.7661053Z dist init r=0, world=2 2022-11-23T03:57:40.7661428Z ok (6.935s) 2022-11-23T03:57:40.7661634Z 2022-11-23T03:57:40.7661943Z ---------------------------------------------------------------------- 2022-11-23T03:57:40.7662277Z Ran 16 tests in 109.975s 2022-11-23T03:57:40.7662549Z 2022-11-23T03:57:40.7662651Z OK 2022-11-23T03:57:40.7662800Z 2022-11-23T03:57:40.7662936Z Generating XML reports... 2022-11-23T03:57:40.7663685Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint/TEST-TestFSDPCheckpoint-20221123035548.xml 2022-11-23T03:57:40.7664088Z 2022-11-23T03:57:40.7664534Z ##[endgroup] 2022-11-23T03:57:40.7665294Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_checkpoint (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_checkpoint_ix96tacd) 2022-11-23T03:57:40.7665708Z 2022-11-23T03:57:40.7666068Z Running distributed/fsdp/test_distributed_checkpoint ... [2022-11-23 03:57:40.732438] 2022-11-23T03:57:40.7666927Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_distributed_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:57:40.733242] 2022-11-23T03:57:54.9110215Z 2022-11-23T03:57:54.9111448Z Expand the folded group to see the log file of distributed/fsdp/test_distributed_checkpoint 2022-11-23T03:57:54.9113803Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_distributed_checkpoint (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_distributed_checkpoint_41lzeto0) 2022-11-23T03:57:54.9116168Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint 2022-11-23T03:57:54.9116931Z 2022-11-23T03:57:54.9117191Z Running tests... 2022-11-23T03:57:54.9118284Z ---------------------------------------------------------------------- 2022-11-23T03:57:54.9119843Z test_distributed_checkpoint_state_dict_type_StateDictType_LOCAL_STATE_DICT (__main__.TestDistributedCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55803 2022-11-23T03:57:54.9121417Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55804 2022-11-23T03:57:54.9123064Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:57:54.9124204Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:57:54.9125736Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:57:54.9126947Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:57:54.9128449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:57:54.9130141Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:57:54.9131286Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:57:54.9132814Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:57:54.9133999Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:57:54.9135167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:57:54.9137499Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:57:54.9139298Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:57:54.9140787Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:57:54.9142092Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:57:54.9143318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:57:54.9144521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:57:54.9145424Z dist init r=1, world=2 2022-11-23T03:57:54.9146038Z dist init r=0, world=2 2022-11-23T03:57:54.9146642Z ok (5.335s) 2022-11-23T03:57:54.9148175Z test_distributed_checkpoint_state_dict_type_StateDictType_SHARDED_STATE_DICT (__main__.TestDistributedCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55950 2022-11-23T03:57:54.9149783Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55951 2022-11-23T03:57:54.9151405Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:57:54.9152562Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:57:54.9154079Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:57:54.9155268Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:57:54.9156405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:57:54.9158053Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:57:54.9159201Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:57:54.9160714Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:57:54.9161904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:57:54.9163038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:57:54.9164720Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:57:54.9166533Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:57:54.9168152Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:57:54.9169471Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:57:54.9170633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:57:54.9171840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:57:54.9172744Z dist init r=0, world=2 2022-11-23T03:57:54.9173360Z dist init r=1, world=2 2022-11-23T03:57:54.9173966Z ok (4.830s) 2022-11-23T03:57:54.9174323Z 2022-11-23T03:57:54.9175039Z ---------------------------------------------------------------------- 2022-11-23T03:57:54.9175874Z Ran 2 tests in 10.166s 2022-11-23T03:57:54.9176266Z 2022-11-23T03:57:54.9176478Z OK 2022-11-23T03:57:54.9176799Z 2022-11-23T03:57:54.9177091Z Generating XML reports... 2022-11-23T03:57:54.9178803Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint/TEST-TestDistributedCheckpoint-20221123035742.xml 2022-11-23T03:57:54.9179771Z 2022-11-23T03:57:54.9180521Z ##[endgroup] 2022-11-23T03:57:54.9182239Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_distributed_checkpoint (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_distributed_checkpoint_41lzeto0) 2022-11-23T03:57:54.9183403Z 2022-11-23T03:57:54.9184119Z Running distributed/elastic/utils/util_test ... [2022-11-23 03:57:54.911066] 2022-11-23T03:57:54.9185938Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/utils/util_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:57:54.911670] 2022-11-23T03:57:59.5068275Z 2022-11-23T03:57:59.5069496Z Expand the folded group to see the log file of distributed/elastic/utils/util_test 2022-11-23T03:57:59.5071853Z ##[group]PRINTING LOG FILE of distributed/elastic/utils/util_test (/var/lib/jenkins/pytorch/test/test-reports/distributed-elastic-utils-util_test_e_24rh0_) 2022-11-23T03:57:59.5074039Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.util_test 2022-11-23T03:57:59.5074770Z 2022-11-23T03:57:59.5075013Z Running tests... 2022-11-23T03:57:59.5076730Z ---------------------------------------------------------------------- 2022-11-23T03:57:59.5077749Z test_get_all_rank_0 (__main__.StoreUtilTest) ... ok (0.591s) 2022-11-23T03:57:59.5078666Z test_get_all_rank_n (__main__.StoreUtilTest) ... ok (0.002s) 2022-11-23T03:57:59.5079584Z test_synchronize (__main__.StoreUtilTest) ... ok (0.003s) 2022-11-23T03:57:59.5080457Z test_get_logger (__main__.UtilTest) ... ok (0.108s) 2022-11-23T03:57:59.5081328Z test_get_logger_custom_name (__main__.UtilTest) ... ok (0.001s) 2022-11-23T03:57:59.5082257Z test_get_logger_different (__main__.UtilTest) ... ok (0.001s) 2022-11-23T03:57:59.5083154Z test_get_logger_none (__main__.UtilTest) ... ok (0.001s) 2022-11-23T03:57:59.5083654Z 2022-11-23T03:57:59.5084384Z ---------------------------------------------------------------------- 2022-11-23T03:57:59.5085211Z Ran 7 tests in 0.708s 2022-11-23T03:57:59.5085603Z 2022-11-23T03:57:59.5085814Z OK 2022-11-23T03:57:59.5086134Z 2022-11-23T03:57:59.5086426Z Generating XML reports... 2022-11-23T03:57:59.5088383Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-StoreUtilTest-20221123035756.xml 2022-11-23T03:57:59.5090347Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-UtilTest-20221123035756.xml 2022-11-23T03:57:59.5091169Z 2022-11-23T03:57:59.5091908Z ##[endgroup] 2022-11-23T03:57:59.5093531Z FINISHED PRINTING LOG FILE of distributed/elastic/utils/util_test (/var/lib/jenkins/pytorch/test/test-reports/distributed-elastic-utils-util_test_e_24rh0_) 2022-11-23T03:57:59.5094438Z 2022-11-23T03:57:59.5095214Z Running distributed/elastic/utils/distributed_test ... [2022-11-23 03:57:59.507153] 2022-11-23T03:57:59.5097116Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/utils/distributed_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:57:59.507767] 2022-11-23T03:58:07.5570902Z 2022-11-23T03:58:07.5571860Z Expand the folded group to see the log file of distributed/elastic/utils/distributed_test 2022-11-23T03:58:07.5574060Z ##[group]PRINTING LOG FILE of distributed/elastic/utils/distributed_test (/var/lib/jenkins/pytorch/test/test-reports/distributed-elastic-utils-distributed_test_uzh_mvfj) 2022-11-23T03:58:07.5576240Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.distributed_test 2022-11-23T03:58:07.5577081Z 2022-11-23T03:58:07.5577336Z Running tests... 2022-11-23T03:58:07.5578423Z ---------------------------------------------------------------------- 2022-11-23T03:58:07.5579458Z test_create_store_multi (__main__.DistributedUtilTest) ... ok (0.616s) 2022-11-23T03:58:07.5580498Z test_create_store_no_port_multi (__main__.DistributedUtilTest) ... ok (0.002s) 2022-11-23T03:58:07.5583557Z test_create_store_single_server (__main__.DistributedUtilTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/66207 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s) 2022-11-23T03:58:07.5585334Z test_create_store_timeout_on_server (__main__.DistributedUtilTest) ... ok (3.040s) 2022-11-23T03:58:07.5586038Z test_create_store_timeout_on_worker (__main__.DistributedUtilTest) ... [E socket.cpp:860] [c10d] The client socket has timed out after 1s while trying to connect to (7ae77914f0c0, 0). 2022-11-23T03:58:07.5586606Z ok (0.002s) 2022-11-23T03:58:07.5587528Z test_port_already_in_use_on_server (__main__.DistributedUtilTest) ... [W socket.cpp:426] [c10d] The server socket has failed to bind to [::]:37505 (errno: 98 - Address already in use). 2022-11-23T03:58:07.5588459Z [W socket.cpp:426] [c10d] The server socket has failed to bind to ?UNKNOWN? (errno: 98 - Address already in use). 2022-11-23T03:58:07.5589091Z [E socket.cpp:462] [c10d] The server socket has failed to listen on any local network address. 2022-11-23T03:58:07.5589548Z ok (0.006s) 2022-11-23T03:58:07.5590276Z test_port_already_in_use_on_worker (__main__.DistributedUtilTest) ... [E socket.cpp:860] [c10d] The client socket has timed out after 1s while trying to connect to (7ae77914f0c0, 57429). 2022-11-23T03:58:07.5590843Z ok (0.007s) 2022-11-23T03:58:07.5591041Z 2022-11-23T03:58:07.5591421Z ---------------------------------------------------------------------- 2022-11-23T03:58:07.5591855Z Ran 7 tests in 3.678s 2022-11-23T03:58:07.5592053Z 2022-11-23T03:58:07.5592182Z OK (skipped=1) 2022-11-23T03:58:07.5592370Z 2022-11-23T03:58:07.5592540Z Generating XML reports... 2022-11-23T03:58:07.5593400Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.distributed_test/TEST-DistributedUtilTest-20221123035801.xml 2022-11-23T03:58:07.5593872Z 2022-11-23T03:58:07.5594388Z ##[endgroup] 2022-11-23T03:58:07.5595281Z FINISHED PRINTING LOG FILE of distributed/elastic/utils/distributed_test (/var/lib/jenkins/pytorch/test/test-reports/distributed-elastic-utils-distributed_test_uzh_mvfj) 2022-11-23T03:58:07.5595737Z 2022-11-23T03:58:07.5596050Z Running distributed/elastic/timer/local_timer_example ... [2022-11-23 03:58:07.557181] 2022-11-23T03:58:07.5596874Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/timer/local_timer_example.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:58:07.557886] 2022-11-23T03:58:26.5475629Z 2022-11-23T03:58:26.5476337Z Expand the folded group to see the log file of distributed/elastic/timer/local_timer_example 2022-11-23T03:58:26.5480722Z ##[group]PRINTING LOG FILE of distributed/elastic/timer/local_timer_example (/var/lib/jenkins/pytorch/test/test-reports/distributed-elastic-timer-local_timer_example_cvp1b5fd) 2022-11-23T03:58:26.5483188Z Test results will be stored in test-reports/python-unittest/distributed.elastic.timer.local_timer_example 2022-11-23T03:58:26.5483967Z 2022-11-23T03:58:26.5484226Z Running tests... 2022-11-23T03:58:26.5485473Z ---------------------------------------------------------------------- 2022-11-23T03:58:26.5487322Z test_example_start_method_spawn (__main__.LocalTimerExample) ... [INFO] 2022-11-23 03:58:09,916 api: Starting LocalTimerServer... max_interval=0.01, daemon=True 2022-11-23T03:58:26.5489116Z [INFO] 2022-11-23 03:58:09,917 api: Starting watchdog thread... 2022-11-23T03:58:26.5490615Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:58:26.5491780Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:58:26.5493342Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:58:26.5494560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:58:26.5495860Z [INFO] 2022-11-23 03:58:11,838 api: Timer client configured to: LocalTimerClient 2022-11-23T03:58:26.5497418Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:58:26.5499383Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:58:26.5500919Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:58:26.5502122Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:58:26.5503421Z [INFO] 2022-11-23 03:58:11,853 api: Timer client configured to: LocalTimerClient 2022-11-23T03:58:26.5504942Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:58:26.5506081Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:58:26.5507607Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:58:26.5508807Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:58:26.5510309Z [INFO] 2022-11-23 03:58:11,993 api: Timer client configured to: LocalTimerClient 2022-11-23T03:58:26.5511876Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:58:26.5513008Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:58:26.5514534Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:58:26.5515715Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:58:26.5516994Z [INFO] 2022-11-23 03:58:12,018 api: Timer client configured to: LocalTimerClient 2022-11-23T03:58:26.5518531Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:58:26.5519666Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:58:26.5521171Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:58:26.5522368Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:58:26.5523655Z [INFO] 2022-11-23 03:58:12,031 api: Timer client configured to: LocalTimerClient 2022-11-23T03:58:26.5525187Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:58:26.5526316Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:58:26.5528637Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:58:26.5530057Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:58:26.5531371Z [INFO] 2022-11-23 03:58:12,031 api: Timer client configured to: LocalTimerClient 2022-11-23T03:58:26.5532916Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:58:26.5534066Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:58:26.5535611Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:58:26.5536775Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:58:26.5538066Z [INFO] 2022-11-23 03:58:12,075 api: Timer client configured to: LocalTimerClient 2022-11-23T03:58:26.5539595Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:58:26.5540721Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:58:26.5542253Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:58:26.5543442Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:58:26.5544721Z [INFO] 2022-11-23 03:58:12,083 api: Timer client configured to: LocalTimerClient 2022-11-23T03:58:26.5546468Z [INFO] 2022-11-23 03:58:12,955 api: Reaping worker_id=[56309]. Expired timers: ['/opt/conda/lib/python3.8/contextlib.py#113'] 2022-11-23T03:58:26.5547826Z [INFO] 2022-11-23 03:58:12,956 api: Successfully reaped worker=[56309] 2022-11-23T03:58:26.5549316Z [INFO] 2022-11-23 03:58:13,140 api: Reaping worker_id=[56307]. Expired timers: ['/opt/conda/lib/python3.8/contextlib.py#113'] 2022-11-23T03:58:26.5550652Z [INFO] 2022-11-23 03:58:13,140 api: Successfully reaped worker=[56307] 2022-11-23T03:58:26.5552131Z [INFO] 2022-11-23 03:58:13,171 api: Reaping worker_id=[56311]. Expired timers: ['/opt/conda/lib/python3.8/contextlib.py#113'] 2022-11-23T03:58:26.5553479Z [INFO] 2022-11-23 03:58:13,171 api: Successfully reaped worker=[56311] 2022-11-23T03:58:26.5554968Z [INFO] 2022-11-23 03:58:13,195 api: Reaping worker_id=[56313]. Expired timers: ['/opt/conda/lib/python3.8/contextlib.py#113'] 2022-11-23T03:58:26.5556312Z [INFO] 2022-11-23 03:58:13,196 api: Successfully reaped worker=[56313] 2022-11-23T03:58:26.5557619Z [INFO] 2022-11-23 03:58:15,263 api: Stopping LocalTimerServer 2022-11-23T03:58:26.5558762Z [INFO] 2022-11-23 03:58:15,263 api: Stopping watchdog thread... 2022-11-23T03:58:26.5559485Z ok (5.960s) 2022-11-23T03:58:26.5560980Z test_torch_mp_example (__main__.LocalTimerExample) ... [INFO] 2022-11-23 03:58:15,274 api: Starting LocalTimerServer... max_interval=0.01, daemon=True 2022-11-23T03:58:26.5562389Z [INFO] 2022-11-23 03:58:15,274 api: Starting watchdog thread... 2022-11-23T03:58:26.5563855Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:58:26.5565001Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:58:26.5566504Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:58:26.5567687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:58:26.5569103Z [INFO] 2022-11-23 03:58:17,318 api: Timer client configured to: LocalTimerClient 2022-11-23T03:58:26.5570648Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:58:26.5571788Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:58:26.5573306Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:58:26.5574490Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:58:26.5575756Z [INFO] 2022-11-23 03:58:17,352 api: Timer client configured to: LocalTimerClient 2022-11-23T03:58:26.5577280Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:58:26.5578410Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:58:26.5579919Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:58:26.5581119Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:58:26.5582395Z [INFO] 2022-11-23 03:58:17,358 api: Timer client configured to: LocalTimerClient 2022-11-23T03:58:26.5583922Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:58:26.5585043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:58:26.5586555Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:58:26.5587734Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:58:26.5589018Z [INFO] 2022-11-23 03:58:17,382 api: Timer client configured to: LocalTimerClient 2022-11-23T03:58:26.5590549Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:58:26.5591693Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:58:26.5593410Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:58:26.5594582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:58:26.5595874Z [INFO] 2022-11-23 03:58:17,394 api: Timer client configured to: LocalTimerClient 2022-11-23T03:58:26.5597399Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:58:26.5598526Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:58:26.5600035Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:58:26.5601205Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:58:26.5602487Z [INFO] 2022-11-23 03:58:17,408 api: Timer client configured to: LocalTimerClient 2022-11-23T03:58:26.5604160Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:58:26.5605310Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:58:26.5606836Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:58:26.5608336Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:58:26.5609637Z [INFO] 2022-11-23 03:58:17,420 api: Timer client configured to: LocalTimerClient 2022-11-23T03:58:26.5611173Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:58:26.5612303Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:58:26.5613812Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:58:26.5614985Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:58:26.5616286Z [INFO] 2022-11-23 03:58:17,431 api: Timer client configured to: LocalTimerClient 2022-11-23T03:58:26.5617395Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:58:26.5617896Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:58:26.5618463Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:58:26.5618909Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:58:26.5619390Z [INFO] 2022-11-23 03:58:23,007 api: Timer client configured to: LocalTimerClient 2022-11-23T03:58:26.5619953Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:58:26.5620379Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:58:26.5620952Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:58:26.5621403Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:58:26.5621880Z [INFO] 2022-11-23 03:58:23,051 api: Timer client configured to: LocalTimerClient 2022-11-23T03:58:26.5622451Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:58:26.5622882Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:58:26.5623442Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:58:26.5623882Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:58:26.5624363Z [INFO] 2022-11-23 03:58:23,109 api: Timer client configured to: LocalTimerClient 2022-11-23T03:58:26.5624934Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:58:26.5625531Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:58:26.5626111Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:58:26.5626555Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:58:26.5627025Z [INFO] 2022-11-23 03:58:23,120 api: Timer client configured to: LocalTimerClient 2022-11-23T03:58:26.5627597Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:58:26.5628019Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:58:26.5628582Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:58:26.5629022Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:58:26.5629559Z [INFO] 2022-11-23 03:58:23,162 api: Timer client configured to: LocalTimerClient 2022-11-23T03:58:26.5630144Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:58:26.5630570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:58:26.5631127Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:58:26.5631570Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:58:26.5632046Z [INFO] 2022-11-23 03:58:23,172 api: Timer client configured to: LocalTimerClient 2022-11-23T03:58:26.5632614Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:58:26.5633045Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:58:26.5633610Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:58:26.5634063Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:58:26.5634536Z [INFO] 2022-11-23 03:58:23,288 api: Timer client configured to: LocalTimerClient 2022-11-23T03:58:26.5635108Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:58:26.5635533Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:58:26.5636104Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:58:26.5636550Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:58:26.5637030Z [INFO] 2022-11-23 03:58:23,291 api: Timer client configured to: LocalTimerClient 2022-11-23T03:58:26.5637600Z [INFO] 2022-11-23 03:58:24,129 api: Reaping worker_id=[57381]. Expired timers: ['/opt/conda/lib/python3.8/contextlib.py#113'] 2022-11-23T03:58:26.5638102Z [INFO] 2022-11-23 03:58:24,129 api: Successfully reaped worker=[57381] 2022-11-23T03:58:26.5638662Z [INFO] 2022-11-23 03:58:24,191 api: Reaping worker_id=[57382]. Expired timers: ['/opt/conda/lib/python3.8/contextlib.py#113'] 2022-11-23T03:58:26.5639162Z [INFO] 2022-11-23 03:58:24,191 api: Successfully reaped worker=[57382] 2022-11-23T03:58:26.5639720Z [INFO] 2022-11-23 03:58:24,242 api: Reaping worker_id=[57384]. Expired timers: ['/opt/conda/lib/python3.8/contextlib.py#113'] 2022-11-23T03:58:26.5640224Z [INFO] 2022-11-23 03:58:24,243 api: Successfully reaped worker=[57384] 2022-11-23T03:58:26.5640782Z [INFO] 2022-11-23 03:58:24,273 api: Reaping worker_id=[57387]. Expired timers: ['/opt/conda/lib/python3.8/contextlib.py#113'] 2022-11-23T03:58:26.5641285Z [INFO] 2022-11-23 03:58:24,274 api: Successfully reaped worker=[57387] 2022-11-23T03:58:26.5641838Z [INFO] 2022-11-23 03:58:24,304 api: Reaping worker_id=[57380]. Expired timers: ['/opt/conda/lib/python3.8/contextlib.py#113'] 2022-11-23T03:58:26.5642380Z [INFO] 2022-11-23 03:58:24,305 local_timer: Process with pid=57380 does not exist. Skipping 2022-11-23T03:58:26.5642918Z [INFO] 2022-11-23 03:58:24,305 api: Successfully reaped worker=[57380] 2022-11-23T03:58:26.5643477Z [INFO] 2022-11-23 03:58:24,305 api: Reaping worker_id=[57383]. Expired timers: ['/opt/conda/lib/python3.8/contextlib.py#113'] 2022-11-23T03:58:26.5644018Z [INFO] 2022-11-23 03:58:24,305 local_timer: Process with pid=57383 does not exist. Skipping 2022-11-23T03:58:26.5644489Z [INFO] 2022-11-23 03:58:24,305 api: Successfully reaped worker=[57383] 2022-11-23T03:58:26.5644922Z [INFO] 2022-11-23 03:58:24,373 api: Stopping LocalTimerServer 2022-11-23T03:58:26.5645344Z [INFO] 2022-11-23 03:58:24,373 api: Stopping watchdog thread... 2022-11-23T03:58:26.5645607Z ok (9.108s) 2022-11-23T03:58:26.5645743Z 2022-11-23T03:58:26.5646012Z ---------------------------------------------------------------------- 2022-11-23T03:58:26.5646330Z Ran 2 tests in 15.068s 2022-11-23T03:58:26.5646478Z 2022-11-23T03:58:26.5646559Z OK 2022-11-23T03:58:26.5646687Z 2022-11-23T03:58:26.5646853Z Generating XML reports... 2022-11-23T03:58:26.5647489Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_example/TEST-LocalTimerExample-20221123035809.xml 2022-11-23T03:58:26.5647886Z 2022-11-23T03:58:26.5648174Z ##[endgroup] 2022-11-23T03:58:26.5648827Z FINISHED PRINTING LOG FILE of distributed/elastic/timer/local_timer_example (/var/lib/jenkins/pytorch/test/test-reports/distributed-elastic-timer-local_timer_example_cvp1b5fd) 2022-11-23T03:58:26.5649203Z 2022-11-23T03:58:26.5649498Z Running distributed/elastic/multiprocessing/api_test ... [2022-11-23 03:58:26.548109] 2022-11-23T03:58:26.5650223Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/multiprocessing/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:58:26.548826] 2022-11-23T03:59:14.5318274Z 2022-11-23T03:59:14.5319638Z Expand the folded group to see the log file of distributed/elastic/multiprocessing/api_test 2022-11-23T03:59:14.5322584Z ##[group]PRINTING LOG FILE of distributed/elastic/multiprocessing/api_test (/var/lib/jenkins/pytorch/test/test-reports/distributed-elastic-multiprocessing-api_test_gbir9117) 2022-11-23T03:59:14.5325638Z Test results will be stored in test-reports/python-unittest/distributed.elastic.multiprocessing.api_test 2022-11-23T03:59:14.5326561Z 2022-11-23T03:59:14.5329409Z Running tests... 2022-11-23T03:59:14.5331037Z ---------------------------------------------------------------------- 2022-11-23T03:59:14.5332096Z test_get_failures (__main__.RunProcResultsTest) ... ok (0.590s) 2022-11-23T03:59:14.5333076Z test_is_failed (__main__.RunProcResultsTest) ... ok (0.001s) 2022-11-23T03:59:14.5334116Z test_args_env_len_mismatch (__main__.StartProcessesListTest) ... ok (0.002s) 2022-11-23T03:59:14.5335226Z test_binary (__main__.StartProcessesListTest) ... hello stderr from 0 2022-11-23T03:59:14.5336391Z hello stdout from 0 2022-11-23T03:59:14.5337425Z hello stderr from 1 2022-11-23T03:59:14.5338465Z hello stdout from 1 2022-11-23T03:59:14.5339471Z ok (0.137s) 2022-11-23T03:59:14.5340664Z test_binary_exit (__main__.StartProcessesListTest) ... bar stderr from 1 2022-11-23T03:59:14.5341710Z bar stdout from 1 2022-11-23T03:59:14.5342894Z failed (exitcode: 138) local_rank: 0 (pid: 57984) of binary: distributed/elastic/multiprocessing/bin/echo1.py 2022-11-23T03:59:14.5344099Z ok (0.141s) 2022-11-23T03:59:14.5344981Z test_binary_incorrect_entrypoint (__main__.StartProcessesListTest) ... ok (0.020s) 2022-11-23T03:59:14.5346165Z test_binary_raises (__main__.StartProcessesListTest) ... Traceback (most recent call last): 2022-11-23T03:59:14.5347299Z File "distributed/elastic/multiprocessing/bin/echo2.py", line 22, in 2022-11-23T03:59:14.5348436Z raise RuntimeError(f"raised from {rank}") 2022-11-23T03:59:14.5349350Z RuntimeError: raised from 0 2022-11-23T03:59:14.5350206Z bar from 1 2022-11-23T03:59:14.5351669Z failed (exitcode: 1) local_rank: 0 (pid: 57987) of binary: distributed/elastic/multiprocessing/bin/echo2.py 2022-11-23T03:59:14.5353503Z ok (0.140s) 2022-11-23T03:59:14.5354615Z test_binary_redirect_and_tee (__main__.StartProcessesListTest) ... world stdout from 1 2022-11-23T03:59:14.5355829Z [trainer0]:hello stdout from 0 2022-11-23T03:59:14.5356736Z [trainer1]:world stderr from 1 2022-11-23T03:59:14.5357826Z ok (1.045s) 2022-11-23T03:59:14.5360072Z test_function (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:59:14.5361816Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:59:14.5363918Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:59:14.5365544Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:59:14.5367528Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:59:14.5369517Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:59:14.5371309Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:59:14.5372512Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:59:14.5373317Z hello stdout from 0 2022-11-23T03:59:14.5373941Z hello stderr from 0 2022-11-23T03:59:14.5374564Z hello stdout from 1 2022-11-23T03:59:14.5375183Z hello stderr from 1 2022-11-23T03:59:14.5375867Z Closing process 57994 via signal SIGTERM 2022-11-23T03:59:14.5376528Z ok (4.129s) 2022-11-23T03:59:14.5378265Z test_function_large_ret_val (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:59:14.5379633Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:59:14.5381174Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:59:14.5382372Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:59:14.5384176Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:59:14.5385300Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:59:14.5386817Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:59:14.5387998Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:59:14.5389524Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:59:14.5390636Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:59:14.5392163Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:59:14.5393356Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:59:14.5394890Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:59:14.5396027Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:59:14.5397542Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:59:14.5398726Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:59:14.5399595Z Closing process 58127 via signal SIGTERM 2022-11-23T03:59:14.5400364Z Closing process 58128 via signal SIGTERM 2022-11-23T03:59:14.5401128Z Closing process 58129 via signal SIGTERM 2022-11-23T03:59:14.5401787Z ok (4.569s) 2022-11-23T03:59:14.5402540Z test_function_raise (__main__.StartProcessesListTest) 2022-11-23T03:59:14.5404373Z run 2x copies of echo2, raise an exception on the first ... /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:59:14.5405878Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:59:14.5407405Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:59:14.5408954Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:59:14.5410514Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:59:14.5411642Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:59:14.5413153Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:59:14.5414334Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:59:14.5415533Z failed (exitcode: 1) local_rank: 0 (pid: 58390) of fn: echo2 (start_method: spawn) 2022-11-23T03:59:14.5416451Z Traceback (most recent call last): 2022-11-23T03:59:14.5417899Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll 2022-11-23T03:59:14.5419026Z self._pc.join(-1) 2022-11-23T03:59:14.5420271Z File "/opt/conda/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 160, in join 2022-11-23T03:59:14.5421060Z raise ProcessRaisedException(msg, error_index, failed_process.pid) 2022-11-23T03:59:14.5421572Z torch.multiprocessing.spawn.ProcessRaisedException: 2022-11-23T03:59:14.5421813Z 2022-11-23T03:59:14.5422031Z -- Process 0 terminated with the following error: 2022-11-23T03:59:14.5422338Z Traceback (most recent call last): 2022-11-23T03:59:14.5422829Z File "/opt/conda/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap 2022-11-23T03:59:14.5423172Z fn(i, *args) 2022-11-23T03:59:14.5423684Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 371, in _wrap 2022-11-23T03:59:14.5424067Z ret = record(fn)(*args_) 2022-11-23T03:59:14.5424604Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper 2022-11-23T03:59:14.5424995Z return f(*args, **kwargs) 2022-11-23T03:59:14.5425373Z File "/var/lib/jenkins/pytorch/test/distributed/elastic/multiprocessing/api_test.py", line 138, in echo2 2022-11-23T03:59:14.5425736Z raise RuntimeError(msg) 2022-11-23T03:59:14.5425989Z RuntimeError: hello 2022-11-23T03:59:14.5426141Z 2022-11-23T03:59:14.5426230Z ok (4.777s) 2022-11-23T03:59:14.5426541Z test_function_with_tensor (__main__.StartProcessesListTest) ... ok (0.005s) 2022-11-23T03:59:14.5426931Z test_invalid_log_dir (__main__.StartProcessesListTest) ... ok (0.002s) 2022-11-23T03:59:14.5427384Z test_multiprocess_context_close (__main__.StartProcessesListTest) ... Closing process 58522 via signal SIGTERM 2022-11-23T03:59:14.5427739Z ok (0.019s) 2022-11-23T03:59:14.5428352Z test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesListTest) ... failed (exitcode: -1) local_rank: 0 (pid: 123) of fn: echo0 (start_method: spawn) 2022-11-23T03:59:14.5428804Z Traceback (most recent call last): 2022-11-23T03:59:14.5429332Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll 2022-11-23T03:59:14.5429748Z self._pc.join(-1) 2022-11-23T03:59:14.5430059Z File "/opt/conda/lib/python3.8/unittest/mock.py", line 1081, in __call__ 2022-11-23T03:59:14.5430397Z return self._mock_call(*args, **kwargs) 2022-11-23T03:59:14.5430738Z File "/opt/conda/lib/python3.8/unittest/mock.py", line 1085, in _mock_call 2022-11-23T03:59:14.5431084Z return self._execute_mock_call(*args, **kwargs) 2022-11-23T03:59:14.5431446Z File "/opt/conda/lib/python3.8/unittest/mock.py", line 1140, in _execute_mock_call 2022-11-23T03:59:14.5431830Z raise effect 2022-11-23T03:59:14.5432166Z torch.multiprocessing.spawn.ProcessRaisedException: test msg 2022-11-23T03:59:14.5432495Z ok (0.011s) 2022-11-23T03:59:14.5433133Z test_pcontext_wait (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:59:14.5433636Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:59:14.5434203Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:59:14.5434652Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:59:14.5434944Z ok (5.509s) 2022-11-23T03:59:14.5435307Z test_subprocess_context_close (__main__.StartProcessesListTest) ... Sending process 58589 closing signal SIGTERM 2022-11-23T03:59:14.5435660Z ok (0.022s) 2022-11-23T03:59:14.5435949Z test_to_map (__main__.StartProcessesListTest) ... ok (0.004s) 2022-11-23T03:59:14.5436382Z test_validate_full_rank (__main__.StartProcessesListTest) ... ok (0.002s) 2022-11-23T03:59:14.5437105Z test_void_function (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:59:14.5437604Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:59:14.5438164Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:59:14.5438615Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:59:14.5439187Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:59:14.5439617Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:59:14.5440181Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:59:14.5440640Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:59:14.5440926Z world 2022-11-23T03:59:14.5441118Z hello 2022-11-23T03:59:14.5441361Z Closing process 58590 via signal SIGTERM 2022-11-23T03:59:14.5441614Z ok (4.229s) 2022-11-23T03:59:14.5441916Z test_args_env_len_mismatch (__main__.StartProcessesTest) ... ok (0.002s) 2022-11-23T03:59:14.5442300Z test_binary_exit (__main__.StartProcessesTest) ... bar stderr from 1 2022-11-23T03:59:14.5442596Z bar stdout from 1 2022-11-23T03:59:14.5442954Z failed (exitcode: 138) local_rank: 0 (pid: 58722) of binary: distributed/elastic/multiprocessing/bin/echo1.py 2022-11-23T03:59:14.5443298Z ok (0.140s) 2022-11-23T03:59:14.5443618Z test_binary_incorrect_entrypoint (__main__.StartProcessesTest) ... ok (0.023s) 2022-11-23T03:59:14.5444037Z test_binary_raises (__main__.StartProcessesTest) ... Traceback (most recent call last): 2022-11-23T03:59:14.5444461Z File "distributed/elastic/multiprocessing/bin/echo2.py", line 22, in 2022-11-23T03:59:14.5444822Z raise RuntimeError(f"raised from {rank}") 2022-11-23T03:59:14.5445098Z RuntimeError: raised from 0 2022-11-23T03:59:14.5445342Z bar from 1 2022-11-23T03:59:14.5445693Z failed (exitcode: 1) local_rank: 0 (pid: 58725) of binary: distributed/elastic/multiprocessing/bin/echo2.py 2022-11-23T03:59:14.5446032Z ok (0.141s) 2022-11-23T03:59:14.5446663Z test_function_large_ret_val (__main__.StartProcessesTest) ... /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:59:14.5447167Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:59:14.5447792Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:59:14.5448240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:59:14.5448897Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:59:14.5449488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:59:14.5450167Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:59:14.5450694Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:59:14.5451367Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:59:14.5451872Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:59:14.5452540Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:59:14.5453081Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:59:14.5453768Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:59:14.5454357Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:59:14.5455041Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:59:14.5455573Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:59:14.5455981Z Closing process 58727 via signal SIGTERM 2022-11-23T03:59:14.5456322Z Closing process 58729 via signal SIGTERM 2022-11-23T03:59:14.5456668Z Closing process 58730 via signal SIGTERM 2022-11-23T03:59:14.5456970Z ok (4.762s) 2022-11-23T03:59:14.5457303Z test_function_raise (__main__.StartProcessesTest) 2022-11-23T03:59:14.5458116Z run 2x copies of echo2, raise an exception on the first ... /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:59:14.5458710Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:59:14.5459394Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:59:14.5459923Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:59:14.5460609Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:59:14.5461112Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:59:14.5461682Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:59:14.5462124Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:59:14.5462517Z failed (exitcode: 1) local_rank: 0 (pid: 58991) of fn: echo2 (start_method: spawn) 2022-11-23T03:59:14.5462849Z Traceback (most recent call last): 2022-11-23T03:59:14.5463373Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll 2022-11-23T03:59:14.5463800Z self._pc.join(-1) 2022-11-23T03:59:14.5464268Z File "/opt/conda/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 160, in join 2022-11-23T03:59:14.5464712Z raise ProcessRaisedException(msg, error_index, failed_process.pid) 2022-11-23T03:59:14.5465135Z torch.multiprocessing.spawn.ProcessRaisedException: 2022-11-23T03:59:14.5465373Z 2022-11-23T03:59:14.5465592Z -- Process 0 terminated with the following error: 2022-11-23T03:59:14.5465890Z Traceback (most recent call last): 2022-11-23T03:59:14.5466383Z File "/opt/conda/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap 2022-11-23T03:59:14.5466726Z fn(i, *args) 2022-11-23T03:59:14.5467231Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 371, in _wrap 2022-11-23T03:59:14.5467611Z ret = record(fn)(*args_) 2022-11-23T03:59:14.5468161Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper 2022-11-23T03:59:14.5468629Z return f(*args, **kwargs) 2022-11-23T03:59:14.5468996Z File "/var/lib/jenkins/pytorch/test/distributed/elastic/multiprocessing/api_test.py", line 138, in echo2 2022-11-23T03:59:14.5469361Z raise RuntimeError(msg) 2022-11-23T03:59:14.5469619Z RuntimeError: hello 2022-11-23T03:59:14.5469769Z 2022-11-23T03:59:14.5469856Z ok (4.187s) 2022-11-23T03:59:14.5470162Z test_function_with_tensor (__main__.StartProcessesTest) ... ok (0.005s) 2022-11-23T03:59:14.5470539Z test_invalid_log_dir (__main__.StartProcessesTest) ... ok (0.002s) 2022-11-23T03:59:14.5470960Z test_multiprocess_context_close (__main__.StartProcessesTest) ... Closing process 59123 via signal SIGTERM 2022-11-23T03:59:14.5471307Z ok (0.016s) 2022-11-23T03:59:14.5471912Z test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesTest) ... failed (exitcode: -1) local_rank: 0 (pid: 123) of fn: echo0 (start_method: spawn) 2022-11-23T03:59:14.5472355Z Traceback (most recent call last): 2022-11-23T03:59:14.5472939Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll 2022-11-23T03:59:14.5473365Z self._pc.join(-1) 2022-11-23T03:59:14.5473680Z File "/opt/conda/lib/python3.8/unittest/mock.py", line 1081, in __call__ 2022-11-23T03:59:14.5474005Z return self._mock_call(*args, **kwargs) 2022-11-23T03:59:14.5474345Z File "/opt/conda/lib/python3.8/unittest/mock.py", line 1085, in _mock_call 2022-11-23T03:59:14.5474691Z return self._execute_mock_call(*args, **kwargs) 2022-11-23T03:59:14.5475050Z File "/opt/conda/lib/python3.8/unittest/mock.py", line 1140, in _execute_mock_call 2022-11-23T03:59:14.5475364Z raise effect 2022-11-23T03:59:14.5475710Z torch.multiprocessing.spawn.ProcessRaisedException: test msg 2022-11-23T03:59:14.5476031Z ok (0.008s) 2022-11-23T03:59:14.5476649Z test_pcontext_wait (__main__.StartProcessesTest) ... /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:59:14.5477143Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:59:14.5477713Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:59:14.5478155Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:59:14.5478451Z ok (5.127s) 2022-11-23T03:59:14.5478811Z test_subprocess_context_close (__main__.StartProcessesTest) ... Sending process 59190 closing signal SIGTERM 2022-11-23T03:59:14.5479146Z ok (0.019s) 2022-11-23T03:59:14.5479432Z test_to_map (__main__.StartProcessesTest) ... ok (0.003s) 2022-11-23T03:59:14.5479793Z test_validate_full_rank (__main__.StartProcessesTest) ... ok (0.001s) 2022-11-23T03:59:14.5480477Z test_void_function (__main__.StartProcessesTest) ... /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:59:14.5480974Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:59:14.5481543Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:59:14.5481993Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:59:14.5482557Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:59:14.5482985Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:59:14.5483550Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:59:14.5483997Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:59:14.5484284Z world 2022-11-23T03:59:14.5484480Z hello 2022-11-23T03:59:14.5484714Z Closing process 59192 via signal SIGTERM 2022-11-23T03:59:14.5484972Z ok (4.156s) 2022-11-23T03:59:14.5485319Z test_from_str_bad_input (__main__.StdTest) ... ok (0.003s) 2022-11-23T03:59:14.5485642Z test_from_value (__main__.StdTest) ... ok (0.003s) 2022-11-23T03:59:14.5485961Z test_from_value_map (__main__.StdTest) ... ok (0.002s) 2022-11-23T03:59:14.5486144Z 2022-11-23T03:59:14.5486421Z ---------------------------------------------------------------------- 2022-11-23T03:59:14.5486723Z Ran 38 tests in 43.964s 2022-11-23T03:59:14.5486872Z 2022-11-23T03:59:14.5486956Z OK 2022-11-23T03:59:14.5487079Z 2022-11-23T03:59:14.5487192Z Generating XML reports... 2022-11-23T03:59:14.5487912Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-RunProcResultsTest-20221123035828.xml 2022-11-23T03:59:14.5488824Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesListTest-20221123035828.xml 2022-11-23T03:59:14.5489889Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesTest-20221123035828.xml 2022-11-23T03:59:14.5490839Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StdTest-20221123035828.xml 2022-11-23T03:59:14.5491238Z 2022-11-23T03:59:14.5491725Z ##[endgroup] 2022-11-23T03:59:14.5492515Z FINISHED PRINTING LOG FILE of distributed/elastic/multiprocessing/api_test (/var/lib/jenkins/pytorch/test/test-reports/distributed-elastic-multiprocessing-api_test_gbir9117) 2022-11-23T03:59:14.5492976Z 2022-11-23T03:59:14.5493298Z Running distributed/elastic/events/lib_test ... [2022-11-23 03:59:14.532565] 2022-11-23T03:59:14.5494009Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/elastic/events/lib_test.py', '-v'] ... [2022-11-23 03:59:14.533182] 2022-11-23T03:59:18.9082704Z 2022-11-23T03:59:18.9083691Z Expand the folded group to see the log file of distributed/elastic/events/lib_test 2022-11-23T03:59:18.9085793Z ##[group]PRINTING LOG FILE of distributed/elastic/events/lib_test (/var/lib/jenkins/pytorch/test/test-reports/distributed-elastic-events-lib_test_n0bj_5hd) 2022-11-23T03:59:18.9087120Z ============================= test session starts ============================== 2022-11-23T03:59:18.9089011Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T03:59:18.9089970Z cachedir: .pytest_cache 2022-11-23T03:59:18.9091622Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T03:59:18.9092987Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T03:59:18.9094516Z plugins: shard-0.1.2, hypothesis-5.35.1, xdist-3.0.2, flakefinder-1.1.0, xdoctest-1.0.2, rerunfailures-10.3 2022-11-23T03:59:18.9095645Z collecting ... collected 8 items 2022-11-23T03:59:18.9098597Z Running 8 items in this shard: test/distributed/elastic/events/lib_test.py::EventLibTest::test_event_created, test/distributed/elastic/events/lib_test.py::EventLibTest::test_event_deser, test/distributed/elastic/events/lib_test.py::EventLibTest::test_get_or_create_logger, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event_does_not_run_if_invalid_dest, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_created, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_deserialize, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_str 2022-11-23T03:59:18.9101033Z 2022-11-23T03:59:18.9101486Z distributed/elastic/events/lib_test.py::EventLibTest::test_event_created PASSED [ 12%] 2022-11-23T03:59:18.9102484Z distributed/elastic/events/lib_test.py::EventLibTest::test_event_deser PASSED [ 25%] 2022-11-23T03:59:18.9103492Z distributed/elastic/events/lib_test.py::EventLibTest::test_get_or_create_logger PASSED [ 37%] 2022-11-23T03:59:18.9104586Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event PASSED [ 50%] 2022-11-23T03:59:18.9106279Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event_does_not_run_if_invalid_dest PASSED [ 62%] 2022-11-23T03:59:18.9107454Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_created PASSED [ 75%] 2022-11-23T03:59:18.9108534Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_deserialize PASSED [ 87%] 2022-11-23T03:59:18.9109603Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_str PASSED [100%] 2022-11-23T03:59:18.9110162Z 2022-11-23T03:59:18.9110464Z ============================== 8 passed in 1.79s =============================== 2022-11-23T03:59:18.9110858Z 2022-11-23T03:59:18.9111486Z ##[endgroup] 2022-11-23T03:59:18.9112922Z FINISHED PRINTING LOG FILE of distributed/elastic/events/lib_test (/var/lib/jenkins/pytorch/test/test-reports/distributed-elastic-events-lib_test_n0bj_5hd) 2022-11-23T03:59:18.9113707Z 2022-11-23T03:59:18.9114531Z Running distributed/checkpoint/test_traverse ... [2022-11-23 03:59:18.908642] 2022-11-23T03:59:18.9116151Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_traverse.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:59:18.909311] 2022-11-23T03:59:23.5602993Z 2022-11-23T03:59:23.5603889Z Expand the folded group to see the log file of distributed/checkpoint/test_traverse 2022-11-23T03:59:23.5606619Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_traverse (/var/lib/jenkins/pytorch/test/test-reports/distributed-checkpoint-test_traverse_8gkm_8uv) 2022-11-23T03:59:23.5609627Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_traverse 2022-11-23T03:59:23.5610539Z 2022-11-23T03:59:23.5610873Z Running tests... 2022-11-23T03:59:23.5612174Z ---------------------------------------------------------------------- 2022-11-23T03:59:23.5613491Z test_get_element (__main__.TestTraverse) ... ok (0.575s) 2022-11-23T03:59:23.5614641Z test_set_element (__main__.TestTraverse) ... ok (0.002s) 2022-11-23T03:59:23.5615873Z test_traverse_doesnt_ignore_intermediate_collections (__main__.TestTraverse) ... ok (0.005s) 2022-11-23T03:59:23.5616970Z test_traverse_nested_dict (__main__.TestTraverse) ... ok (0.001s) 2022-11-23T03:59:23.5618183Z test_traverse_nested_list (__main__.TestTraverse) ... ok (0.002s) 2022-11-23T03:59:23.5619328Z test_traverse_shallow (__main__.TestTraverse) ... ok (0.002s) 2022-11-23T03:59:23.5620532Z test_traverse_with_ordered_dict (__main__.TestTraverse) ... ok (0.001s) 2022-11-23T03:59:23.5621120Z 2022-11-23T03:59:23.5621893Z ---------------------------------------------------------------------- 2022-11-23T03:59:23.5622734Z Ran 7 tests in 0.590s 2022-11-23T03:59:23.5623141Z 2022-11-23T03:59:23.5623359Z OK 2022-11-23T03:59:23.5623683Z 2022-11-23T03:59:23.5623983Z Generating XML reports... 2022-11-23T03:59:23.5625585Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_traverse/TEST-TestTraverse-20221123035920.xml 2022-11-23T03:59:23.5626454Z 2022-11-23T03:59:23.5627180Z ##[endgroup] 2022-11-23T03:59:23.5628843Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_traverse (/var/lib/jenkins/pytorch/test/test-reports/distributed-checkpoint-test_traverse_8gkm_8uv) 2022-11-23T03:59:23.5629767Z 2022-11-23T03:59:23.5630636Z Running distributed/checkpoint/test_file_system_checkpoint_cpu ... [2022-11-23 03:59:23.560814] 2022-11-23T03:59:23.5632701Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_file_system_checkpoint_cpu.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:59:23.561455] 2022-11-23T03:59:40.9464696Z 2022-11-23T03:59:40.9466374Z Expand the folded group to see the log file of distributed/checkpoint/test_file_system_checkpoint_cpu 2022-11-23T03:59:40.9469686Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_file_system_checkpoint_cpu (/var/lib/jenkins/pytorch/test/test-reports/distributed-checkpoint-test_file_system_checkpoint_cpu_tccnllzr) 2022-11-23T03:59:40.9473256Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu 2022-11-23T03:59:40.9474087Z 2022-11-23T03:59:40.9474358Z Running tests... 2022-11-23T03:59:40.9475500Z ---------------------------------------------------------------------- 2022-11-23T03:59:40.9478768Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84440 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.596s) 2022-11-23T03:59:40.9483865Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84531 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.003s) 2022-11-23T03:59:40.9486841Z test_save_load_bytes (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59523 2022-11-23T03:59:40.9488455Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59524 2022-11-23T03:59:40.9490432Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:59:40.9491608Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:59:40.9493761Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:59:40.9496013Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:59:40.9497881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:59:40.9500851Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:59:40.9502833Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:59:40.9505500Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:59:40.9508576Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:59:40.9510871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:59:40.9513621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:59:40.9516270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:59:40.9520394Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:59:40.9525652Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:59:40.9528030Z ok (4.233s) 2022-11-23T03:59:40.9529295Z test_switch_between_sharded_tensor_to_tensor (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59663 2022-11-23T03:59:40.9530784Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59664 2022-11-23T03:59:40.9532482Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:59:40.9533654Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:59:40.9535218Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:59:40.9536444Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:59:40.9538033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:59:40.9539727Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:59:40.9540877Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:59:40.9542440Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:59:40.9543643Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:59:40.9544765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:59:40.9546024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:59:40.9547269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:59:40.9549190Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:59:40.9551052Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:59:40.9552059Z ok (4.529s) 2022-11-23T03:59:40.9552970Z test_read_write_only_tensor (__main__.TestDistributedStateDictSaveLoad) ... ok (0.045s) 2022-11-23T03:59:40.9554616Z test_read_write_shard_tensor (__main__.TestDistributedStateDictSaveLoadWithSharedTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59803 2022-11-23T03:59:40.9556188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59804 2022-11-23T03:59:40.9557803Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:59:40.9558955Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:59:40.9560515Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:59:40.9561717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:59:40.9562838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:59:40.9564491Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:59:40.9565640Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:59:40.9567184Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:59:40.9568733Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:59:40.9569874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:59:40.9571146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:59:40.9572414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:59:40.9574187Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:59:40.9575860Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:59:40.9576326Z ok (4.024s) 2022-11-23T03:59:40.9576493Z 2022-11-23T03:59:40.9576752Z ---------------------------------------------------------------------- 2022-11-23T03:59:40.9577072Z Ran 6 tests in 13.431s 2022-11-23T03:59:40.9577225Z 2022-11-23T03:59:40.9577323Z OK (skipped=2) 2022-11-23T03:59:40.9577467Z 2022-11-23T03:59:40.9577584Z Generating XML reports... 2022-11-23T03:59:40.9578265Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedReshardOnLoad-20221123035925.xml 2022-11-23T03:59:40.9579271Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoad-20221123035925.xml 2022-11-23T03:59:40.9580269Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20221123035925.xml 2022-11-23T03:59:40.9580732Z 2022-11-23T03:59:40.9581063Z ##[endgroup] 2022-11-23T03:59:40.9581742Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_file_system_checkpoint_cpu (/var/lib/jenkins/pytorch/test/test-reports/distributed-checkpoint-test_file_system_checkpoint_cpu_tccnllzr) 2022-11-23T03:59:40.9582144Z 2022-11-23T03:59:40.9582430Z Running distributed/checkpoint/test_dedup_tensors ... [2022-11-23 03:59:40.946804] 2022-11-23T03:59:40.9583208Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_dedup_tensors.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:59:40.947467] 2022-11-23T03:59:45.8588173Z 2022-11-23T03:59:45.8589447Z Expand the folded group to see the log file of distributed/checkpoint/test_dedup_tensors 2022-11-23T03:59:45.8592154Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_dedup_tensors (/var/lib/jenkins/pytorch/test/test-reports/distributed-checkpoint-test_dedup_tensors_1maxlju0) 2022-11-23T03:59:45.8594881Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_dedup_tensors 2022-11-23T03:59:45.8595785Z 2022-11-23T03:59:45.8596115Z Running tests... 2022-11-23T03:59:45.8597498Z ---------------------------------------------------------------------- 2022-11-23T03:59:45.8598517Z test_dedup_shards (__main__.TestDedupTensor) ... ok (0.592s) 2022-11-23T03:59:45.8599076Z 2022-11-23T03:59:45.8600018Z ---------------------------------------------------------------------- 2022-11-23T03:59:45.8601055Z Ran 1 test in 0.593s 2022-11-23T03:59:45.8601540Z 2022-11-23T03:59:45.8601836Z OK 2022-11-23T03:59:45.8602249Z 2022-11-23T03:59:45.8602613Z Generating XML reports... 2022-11-23T03:59:45.8604277Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_dedup_tensors/TEST-TestDedupTensor-20221123035942.xml 2022-11-23T03:59:45.8605171Z 2022-11-23T03:59:45.8605903Z ##[endgroup] 2022-11-23T03:59:45.8607576Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_dedup_tensors (/var/lib/jenkins/pytorch/test/test-reports/distributed-checkpoint-test_dedup_tensors_1maxlju0) 2022-11-23T03:59:45.8608664Z 2022-11-23T03:59:45.8609396Z Running distributed/algorithms/test_join ... [2022-11-23 03:59:45.859044] 2022-11-23T03:59:45.8611205Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/test_join.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:59:45.859705] 2022-11-23T04:00:32.4396027Z 2022-11-23T04:00:32.4397245Z Expand the folded group to see the log file of distributed/algorithms/test_join 2022-11-23T04:00:32.4399454Z ##[group]PRINTING LOG FILE of distributed/algorithms/test_join (/var/lib/jenkins/pytorch/test/test-reports/distributed-algorithms-test_join_phh60fjl) 2022-11-23T04:00:32.4401693Z Test results will be stored in test-reports/python-unittest/distributed.algorithms.test_join 2022-11-23T04:00:32.4404860Z 2022-11-23T04:00:32.4405756Z Running tests... 2022-11-23T04:00:32.4409519Z ---------------------------------------------------------------------- 2022-11-23T04:00:32.4410626Z test_join_kwargs (__main__.TestJoin) 2022-11-23T04:00:32.4411976Z Tests passing keyword arguments to the context manager. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60076 2022-11-23T04:00:32.4413347Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60077 2022-11-23T04:00:32.4415244Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:00:32.4416490Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:00:32.4419010Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:00:32.4420245Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:00:32.4421341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:00:32.4422622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:00:32.4424490Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:00:32.4425659Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:00:32.4427216Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:00:32.4428408Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:00:32.4429775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:00:32.4431019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:00:32.4432765Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:00:32.4434593Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:00:32.4436071Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:00:32.4437384Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:00:32.4438199Z ok (5.020s) 2022-11-23T04:00:32.4438935Z test_multiple_joinable_disable (__main__.TestJoin) 2022-11-23T04:00:32.4440146Z Tests ``enable=False`` for multiple :class:`Joinable` s. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60219 2022-11-23T04:00:32.4441478Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60220 2022-11-23T04:00:32.4443109Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:00:32.4444230Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:00:32.4445752Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:00:32.4446947Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:00:32.4448392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:00:32.4449638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:00:32.4451311Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:00:32.4452471Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:00:32.4454002Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:00:32.4455199Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:00:32.4456313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:00:32.4457540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:00:32.4459257Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:00:32.4461063Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:00:32.4462538Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:00:32.4463771Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:00:32.4464541Z ok (4.529s) 2022-11-23T04:00:32.4465052Z test_multiple_joinables (__main__.TestJoin) 2022-11-23T04:00:32.4466277Z Tests the main hooks and post-hooks of multiple :class:`Joinable` s ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60362 2022-11-23T04:00:32.4467269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60363 2022-11-23T04:00:32.4468426Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:00:32.4469242Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:00:32.4470331Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:00:32.4471187Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:00:32.4472105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:00:32.4473001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:00:32.4474189Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:00:32.4475009Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:00:32.4476108Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:00:32.4476968Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:00:32.4477754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:00:32.4478638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:00:32.4479882Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:00:32.4481202Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:00:32.4482269Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:00:32.4483204Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:00:32.4483805Z ok (5.130s) 2022-11-23T04:00:32.4484339Z test_multiple_joinables_throw (__main__.TestJoin) 2022-11-23T04:00:32.4485220Z Tests ``throw_on_early_termination=True`` for multiple ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60505 2022-11-23T04:00:32.4486166Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60506 2022-11-23T04:00:32.4487330Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:00:32.4488240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:00:32.4489347Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:00:32.4490221Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:00:32.4491036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:00:32.4491928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:00:32.4493135Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:00:32.4493964Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:00:32.4495069Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:00:32.4495917Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:00:32.4496869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:00:32.4497761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:00:32.4499022Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:00:32.4500336Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:00:32.4501407Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:00:32.4502357Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:00:32.4502946Z ok (4.727s) 2022-11-23T04:00:32.4503436Z test_single_joinable (__main__.TestJoin) 2022-11-23T04:00:32.4504623Z Tests the main hooks and post-hooks of a single :class:`Joinable` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60648 2022-11-23T04:00:32.4505819Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60649 2022-11-23T04:00:32.4507018Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:00:32.4507842Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:00:32.4508941Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:00:32.4509808Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:00:32.4510595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:00:32.4511487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:00:32.4512686Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:00:32.4513522Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:00:32.4514637Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:00:32.4515495Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:00:32.4516294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:00:32.4517167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:00:32.4518424Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:00:32.4519717Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:00:32.4520781Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:00:32.4521739Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:00:32.4522335Z ok (4.427s) 2022-11-23T04:00:32.4523000Z test_single_joinable_disable (__main__.TestJoin) 2022-11-23T04:00:32.4523869Z Tests ``enable=False`` for a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60791 2022-11-23T04:00:32.4524815Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60792 2022-11-23T04:00:32.4525972Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:00:32.4526804Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:00:32.4528057Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:00:32.4528924Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:00:32.4529729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:00:32.4530786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:00:32.4532022Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:00:32.4532846Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:00:32.4533936Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:00:32.4534801Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:00:32.4535618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:00:32.4536504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:00:32.4537743Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:00:32.4539160Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:00:32.4540239Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:00:32.4541169Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:00:32.4541763Z ok (4.427s) 2022-11-23T04:00:32.4542298Z test_single_joinable_main_hooks (__main__.TestJoin) 2022-11-23T04:00:32.4543175Z Tests the main hooks of a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60934 2022-11-23T04:00:32.4544123Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60935 2022-11-23T04:00:32.4545303Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:00:32.4546129Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:00:32.4547230Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:00:32.4548098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:00:32.4548908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:00:32.4549796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:00:32.4551002Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:00:32.4551830Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:00:32.4552929Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:00:32.4553776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:00:32.4554583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:00:32.4555493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:00:32.4556744Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:00:32.4558054Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:00:32.4559144Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:00:32.4560094Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:00:32.4560690Z ok (4.828s) 2022-11-23T04:00:32.4561219Z test_single_joinable_post_hooks (__main__.TestJoin) 2022-11-23T04:00:32.4562401Z Tests the post-hooks of a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61077 2022-11-23T04:00:32.4563488Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61078 2022-11-23T04:00:32.4564660Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:00:32.4565493Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:00:32.4566601Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:00:32.4567461Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:00:32.4568517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:00:32.4569419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:00:32.4570633Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:00:32.4571598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:00:32.4572751Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:00:32.4573612Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:00:32.4574424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:00:32.4575305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:00:32.4576550Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:00:32.4577856Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:00:32.4578931Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:00:32.4579886Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:00:32.4580497Z ok (4.826s) 2022-11-23T04:00:32.4581013Z test_single_joinable_throw (__main__.TestJoin) 2022-11-23T04:00:32.4581855Z Tests ``throw_on_early_termination=True`` for a single ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61220 2022-11-23T04:00:32.4582806Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61221 2022-11-23T04:00:32.4583749Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:00:32.4584270Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:00:32.4584963Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:00:32.4585510Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:00:32.4586023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:00:32.4586579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:00:32.4587348Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:00:32.4587872Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:00:32.4588569Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:00:32.4589116Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:00:32.4589633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:00:32.4590198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:00:32.4590980Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:00:32.4591881Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:00:32.4592567Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:00:32.4593171Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:00:32.4593516Z ok (4.734s) 2022-11-23T04:00:32.4593659Z 2022-11-23T04:00:32.4593936Z ---------------------------------------------------------------------- 2022-11-23T04:00:32.4594262Z Ran 9 tests in 42.651s 2022-11-23T04:00:32.4594423Z 2022-11-23T04:00:32.4594511Z OK 2022-11-23T04:00:32.4594624Z 2022-11-23T04:00:32.4594738Z Generating XML reports... 2022-11-23T04:00:32.4595311Z Generated XML report: test-reports/python-unittest/distributed.algorithms.test_join/TEST-TestJoin-20221123035947.xml 2022-11-23T04:00:32.4595620Z 2022-11-23T04:00:32.4596355Z ##[endgroup] 2022-11-23T04:00:32.4597044Z FINISHED PRINTING LOG FILE of distributed/algorithms/test_join (/var/lib/jenkins/pytorch/test/test-reports/distributed-algorithms-test_join_phh60fjl) 2022-11-23T04:00:32.4597392Z 2022-11-23T04:00:32.4597664Z Running distributed/_tensor/test_view_ops ... [2022-11-23 04:00:32.439784] 2022-11-23T04:00:32.4598353Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_view_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 04:00:32.440456] 2022-11-23T04:00:46.2832943Z 2022-11-23T04:00:46.2833631Z Expand the folded group to see the log file of distributed/_tensor/test_view_ops 2022-11-23T04:00:46.2835772Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_view_ops (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_view_ops_9g5t1o7y) 2022-11-23T04:00:46.2837903Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_view_ops 2022-11-23T04:00:46.2838593Z 2022-11-23T04:00:46.2838868Z Running tests... 2022-11-23T04:00:46.2839945Z ---------------------------------------------------------------------- 2022-11-23T04:00:46.2841237Z test_view_groups (__main__.TestViewOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61430 2022-11-23T04:00:46.2842731Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61431 2022-11-23T04:00:46.2843898Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 61432 2022-11-23T04:00:46.2845047Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 61433 2022-11-23T04:00:46.2846200Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 61434 2022-11-23T04:00:46.2847350Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 61435 2022-11-23T04:00:46.2849233Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:00:46.2850403Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:00:46.2851949Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:00:46.2853159Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:00:46.2854296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:00:46.2855952Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:00:46.2857095Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:00:46.2858636Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:00:46.2859843Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:00:46.2860934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:00:46.2862575Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:00:46.2864182Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:00:46.2865722Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:00:46.2866906Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:00:46.2868014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:00:46.2869651Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:00:46.2870803Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:00:46.2872311Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:00:46.2873508Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:00:46.2874843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:00:46.2876504Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:00:46.2877656Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:00:46.2879188Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:00:46.2880389Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:00:46.2881480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T04:00:46.2883126Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:00:46.2884268Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:00:46.2885808Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:00:46.2887011Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:00:46.2888519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T04:00:46.2889415Z ok (5.044s) 2022-11-23T04:00:46.2890423Z test_view_ops (__main__.TestViewOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61832 2022-11-23T04:00:46.2891691Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61833 2022-11-23T04:00:46.2892846Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 61834 2022-11-23T04:00:46.2894003Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 61835 2022-11-23T04:00:46.2895157Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 61836 2022-11-23T04:00:46.2896302Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 61837 2022-11-23T04:00:46.2897677Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:00:46.2898492Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:00:46.2899585Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:00:46.2900454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:00:46.2901260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T04:00:46.2902432Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:00:46.2903254Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:00:46.2904361Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:00:46.2905209Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:00:46.2906154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T04:00:46.2907343Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:00:46.2908166Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:00:46.2909262Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:00:46.2910123Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:00:46.2910922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:00:46.2912182Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:00:46.2913078Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:00:46.2914407Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:00:46.2915358Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:00:46.2916242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:00:46.2917549Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:00:46.2918450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:00:46.2919661Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:00:46.2920593Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:00:46.2921482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:00:46.2922791Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:00:46.2923698Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:00:46.2924906Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:00:46.2925845Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:00:46.2926740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:00:46.2927489Z skip: Need at least 6 CUDA devices (4.767s) 2022-11-23T04:00:46.2927927Z 2022-11-23T04:00:46.2928500Z ---------------------------------------------------------------------- 2022-11-23T04:00:46.2929182Z Ran 2 tests in 9.812s 2022-11-23T04:00:46.2929499Z 2022-11-23T04:00:46.2929698Z OK (skipped=1) 2022-11-23T04:00:46.2929998Z 2022-11-23T04:00:46.2930231Z Generating XML reports... 2022-11-23T04:00:46.2931435Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_view_ops/TEST-TestViewOps-20221123040034.xml 2022-11-23T04:00:46.2932092Z 2022-11-23T04:00:46.2932790Z ##[endgroup] 2022-11-23T04:00:46.2934034Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_view_ops (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_view_ops_9g5t1o7y) 2022-11-23T04:00:46.2934725Z 2022-11-23T04:00:46.2935285Z Running distributed/_tensor/test_tensor_ops ... [2022-11-23 04:00:46.283527] 2022-11-23T04:00:46.2936784Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_tensor_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 04:00:46.284204] 2022-11-23T04:02:14.1955761Z 2022-11-23T04:02:14.1956762Z Expand the folded group to see the log file of distributed/_tensor/test_tensor_ops 2022-11-23T04:02:14.1961054Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_tensor_ops (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_tensor_ops_ethy31lm) 2022-11-23T04:02:14.1964018Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_tensor_ops 2022-11-23T04:02:14.1964729Z 2022-11-23T04:02:14.1965009Z Running tests... 2022-11-23T04:02:14.1966126Z ---------------------------------------------------------------------- 2022-11-23T04:02:14.1967430Z test_aten_contiguous (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62301 2022-11-23T04:02:14.1969188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62302 2022-11-23T04:02:14.1970878Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:14.1972068Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:14.1973632Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:14.1974888Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:14.1976320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:02:14.1977590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:02:14.1979320Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:14.1980491Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:14.1982054Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:14.1983269Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:14.1984428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:02:14.1985668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:02:14.1987441Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:14.1989291Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:14.1990803Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:14.1992140Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:14.1993456Z ok (5.307s) 2022-11-23T04:02:14.1994501Z test_clone (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62444 2022-11-23T04:02:14.1995786Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62445 2022-11-23T04:02:14.1997410Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:14.1998557Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:14.2000124Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:14.2001328Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:14.2002444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:02:14.2003688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:02:14.2005377Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:14.2006506Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:14.2008160Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:14.2009361Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:14.2010706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:02:14.2011946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:02:14.2013710Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:14.2015539Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:14.2017013Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:14.2018331Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:14.2019148Z ok (4.630s) 2022-11-23T04:02:14.2020213Z test_contiguous (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62591 2022-11-23T04:02:14.2021541Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62592 2022-11-23T04:02:14.2023343Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:14.2024514Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:14.2027079Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:14.2028944Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:14.2030757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:02:14.2032019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:02:14.2033750Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:14.2034898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:14.2036475Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:14.2037670Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:14.2038768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:02:14.2040005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:02:14.2041736Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:14.2043564Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:14.2045041Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:14.2046344Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:14.2047165Z ok (4.528s) 2022-11-23T04:02:14.2048513Z test_detach (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62742 2022-11-23T04:02:14.2075570Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62743 2022-11-23T04:02:14.2077352Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:14.2078341Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:14.2079645Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:14.2080665Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:14.2081601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:02:14.2082697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:02:14.2084564Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:14.2085526Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:14.2086837Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:14.2088158Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:14.2089194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:02:14.2090412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:02:14.2092145Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:14.2093962Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:14.2095687Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:14.2097027Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:14.2097841Z ok (4.533s) 2022-11-23T04:02:14.2098872Z test_empty_like (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62885 2022-11-23T04:02:14.2100169Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62886 2022-11-23T04:02:14.2101615Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:14.2102490Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:14.2103654Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:14.2104558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:14.2105411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:02:14.2106331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:02:14.2107577Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:14.2108445Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:14.2109600Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:14.2110499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:14.2111336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:02:14.2112265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:02:14.2113561Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:14.2114930Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:14.2116053Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:14.2117036Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:14.2117646Z ok (4.433s) 2022-11-23T04:02:14.2118445Z test_fill_inplace (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63028 2022-11-23T04:02:14.2119438Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63029 2022-11-23T04:02:14.2120641Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:14.2121501Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:14.2122667Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:14.2123706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:14.2124548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:02:14.2125471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:02:14.2126736Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:14.2127598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:14.2129161Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:14.2130098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:14.2131114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:02:14.2132088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:02:14.2133454Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:14.2134868Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:14.2136024Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:14.2137031Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:14.2137679Z ok (4.625s) 2022-11-23T04:02:14.2138537Z test_fill_inplace_partial_sum (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63171 2022-11-23T04:02:14.2139599Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63172 2022-11-23T04:02:14.2140902Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:14.2141771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:14.2142865Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:14.2143745Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:14.2144543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:02:14.2145420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:02:14.2146597Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:14.2147410Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:14.2148510Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:14.2149362Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:14.2150151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:02:14.2151005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:02:14.2152225Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:14.2153507Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:14.2154598Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:14.2155531Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:14.2156115Z ok (5.133s) 2022-11-23T04:02:14.2156866Z test_full_like (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63314 2022-11-23T04:02:14.2157912Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63315 2022-11-23T04:02:14.2159076Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:14.2159899Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:14.2160990Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:14.2161848Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:14.2162645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:02:14.2163532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:02:14.2164799Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:14.2165641Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:14.2166732Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:14.2167588Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:14.2168506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:02:14.2169383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:02:14.2170622Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:14.2171932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:14.2172988Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:14.2173923Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:14.2174512Z ok (4.524s) 2022-11-23T04:02:14.2175343Z test_index (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63457 2022-11-23T04:02:14.2176351Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63458 2022-11-23T04:02:14.2177488Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:14.2178299Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:14.2179354Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:14.2180189Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:14.2180977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:02:14.2181866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:02:14.2183041Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:14.2183878Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:14.2184965Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:14.2185797Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:14.2186582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:02:14.2187460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:02:14.2188683Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:14.2190114Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:14.2191169Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:14.2192098Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:14.2192666Z ok (13.751s) 2022-11-23T04:02:14.2193426Z test_inplace_op (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63666 2022-11-23T04:02:14.2194352Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63667 2022-11-23T04:02:14.2195500Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:14.2196311Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:14.2197510Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:14.2198379Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:14.2199160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:02:14.2200052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:02:14.2201234Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:14.2202040Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:14.2203117Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:14.2203960Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:14.2204744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:02:14.2205626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:02:14.2206834Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:14.2208671Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:14.2209755Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:14.2210700Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:14.2211297Z ok (5.131s) 2022-11-23T04:02:14.2212048Z test_ones_like (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63813 2022-11-23T04:02:14.2212979Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63814 2022-11-23T04:02:14.2214130Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:14.2214948Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:14.2216033Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:14.2216883Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:14.2217672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:02:14.2218544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:02:14.2219717Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:14.2220513Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:14.2221550Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:14.2222181Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:14.2222606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:02:14.2223064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:02:14.2223706Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:14.2224374Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:14.2224923Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:14.2225401Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:14.2225705Z ok (4.530s) 2022-11-23T04:02:14.2226168Z test_ones_like_partial_sum (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63956 2022-11-23T04:02:14.2226677Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63957 2022-11-23T04:02:14.2227281Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:14.2227710Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:14.2228278Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:14.2228718Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:14.2229135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:02:14.2229590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:02:14.2230225Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:14.2230657Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:14.2231234Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:14.2231681Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:14.2232093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:02:14.2232555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:02:14.2233203Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:14.2233869Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:14.2234424Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:14.2234924Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:14.2235226Z ok (4.428s) 2022-11-23T04:02:14.2235624Z test_op_out_variant (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64099 2022-11-23T04:02:14.2236121Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64100 2022-11-23T04:02:14.2236720Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:14.2237145Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:14.2237709Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:14.2238151Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:14.2238567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:02:14.2239086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:02:14.2239693Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:14.2240121Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:14.2240687Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:14.2241143Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:14.2241558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:02:14.2242024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:02:14.2242661Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:14.2243382Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:14.2243943Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:14.2244427Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:14.2244742Z ok (4.529s) 2022-11-23T04:02:14.2245143Z test_zero_inplace (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64246 2022-11-23T04:02:14.2245641Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64247 2022-11-23T04:02:14.2246248Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:14.2246683Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:14.2247257Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:14.2247785Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:14.2248199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:02:14.2248656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:02:14.2249344Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:14.2249848Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:14.2250526Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:14.2251048Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:14.2251553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:02:14.2252115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:02:14.2252886Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:14.2253689Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:14.2254356Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:14.2254943Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:14.2255294Z ok (4.430s) 2022-11-23T04:02:14.2255771Z test_zeros_like (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64389 2022-11-23T04:02:14.2256353Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64390 2022-11-23T04:02:14.2257076Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:14.2257714Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:14.2258408Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:14.2258955Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:14.2259451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:02:14.2260013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:02:14.2260753Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:14.2261277Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:14.2261941Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:14.2262458Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:14.2262887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:02:14.2263356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:02:14.2263992Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:14.2264675Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:14.2265234Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:14.2265729Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:14.2266039Z ok (4.432s) 2022-11-23T04:02:14.2266459Z test_zeros_like_partial_sum (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64532 2022-11-23T04:02:14.2266974Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64533 2022-11-23T04:02:14.2267570Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:14.2268005Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:14.2268579Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:14.2269031Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:14.2269453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:02:14.2270062Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:14.2270495Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:14.2271065Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:14.2271515Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:14.2271934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:02:14.2272392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:02:14.2272859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:02:14.2273496Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:14.2274164Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:14.2274718Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:14.2275266Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:14.2275575Z ok (4.927s) 2022-11-23T04:02:14.2275715Z 2022-11-23T04:02:14.2275987Z ---------------------------------------------------------------------- 2022-11-23T04:02:14.2276302Z Ran 16 tests in 83.876s 2022-11-23T04:02:14.2276451Z 2022-11-23T04:02:14.2276533Z OK 2022-11-23T04:02:14.2276656Z 2022-11-23T04:02:14.2276767Z Generating XML reports... 2022-11-23T04:02:14.2277348Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_tensor_ops/TEST-DistTensorOpsTest-20221123040048.xml 2022-11-23T04:02:14.2277674Z 2022-11-23T04:02:14.2278099Z ##[endgroup] 2022-11-23T04:02:14.2278695Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_tensor_ops (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_tensor_ops_ethy31lm) 2022-11-23T04:02:14.2279028Z 2022-11-23T04:02:14.2279302Z Running distributed/_tensor/test_pointwise_ops ... [2022-11-23 04:02:14.196414] 2022-11-23T04:02:14.2280062Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_pointwise_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 04:02:14.197155] 2022-11-23T04:02:37.7151838Z 2022-11-23T04:02:37.7152588Z Expand the folded group to see the log file of distributed/_tensor/test_pointwise_ops 2022-11-23T04:02:37.7157647Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_pointwise_ops (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_pointwise_ops_5h27fcew) 2022-11-23T04:02:37.7161993Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_pointwise_ops 2022-11-23T04:02:37.7162805Z 2022-11-23T04:02:37.7163049Z Running tests... 2022-11-23T04:02:37.7164157Z ---------------------------------------------------------------------- 2022-11-23T04:02:37.7165504Z test_activations (__main__.DistElementwiseOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64742 2022-11-23T04:02:37.7166904Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64743 2022-11-23T04:02:37.7168769Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:37.7169982Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:37.7171555Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:37.7172769Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:37.7173900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:02:37.7175500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:02:37.7177759Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:37.7179200Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:37.7181040Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:37.7182250Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:37.7183367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:02:37.7184601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:02:37.7186306Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:37.7188122Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:37.7189619Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:37.7190916Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:37.7192255Z ok (5.916s) 2022-11-23T04:02:37.7193373Z test_dropout (__main__.DistElementwiseOpsTest) ... skip: testing RNG based ops is broken: https://github.com/pytorch/tau/issues/494 (0.003s) 2022-11-23T04:02:37.7194936Z test_dropout_backward (__main__.DistElementwiseOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64889 2022-11-23T04:02:37.7196288Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64890 2022-11-23T04:02:37.7197903Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:37.7199054Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:37.7200589Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:37.7201799Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:37.7203282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:02:37.7204530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:02:37.7206209Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:37.7207333Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:37.7209075Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:37.7210272Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:37.7211387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:02:37.7212608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:02:37.7214364Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:37.7216184Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:37.7217665Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:37.7218953Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:37.7219766Z ok (4.525s) 2022-11-23T04:02:37.7220876Z test_dropout_errors (__main__.DistElementwiseOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65036 2022-11-23T04:02:37.7222213Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65037 2022-11-23T04:02:37.7223818Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:37.7224957Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:37.7226499Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:37.7227660Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:37.7228869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:02:37.7230100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:02:37.7231744Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:37.7232882Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:37.7234409Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:37.7235609Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:37.7236730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:02:37.7238135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:02:37.7239865Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:37.7241674Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:37.7243161Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:37.7244469Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:37.7245285Z ok (4.533s) 2022-11-23T04:02:37.7246356Z test_mul_out (__main__.DistElementwiseOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65179 2022-11-23T04:02:37.7247658Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65180 2022-11-23T04:02:37.7249857Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:37.7251019Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:37.7252566Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:37.7253756Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:37.7254870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:02:37.7256109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:02:37.7257781Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:37.7258566Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:37.7259224Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:37.7259688Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:37.7260110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:02:37.7260575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:02:37.7261219Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:37.7261902Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:37.7262451Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:37.7262950Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:37.7263269Z ok (4.533s) 2022-11-23T04:02:37.7263408Z 2022-11-23T04:02:37.7263691Z ---------------------------------------------------------------------- 2022-11-23T04:02:37.7264009Z Ran 5 tests in 19.512s 2022-11-23T04:02:37.7264162Z 2022-11-23T04:02:37.7264260Z OK (skipped=1) 2022-11-23T04:02:37.7264400Z 2022-11-23T04:02:37.7264516Z Generating XML reports... 2022-11-23T04:02:37.7265126Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_pointwise_ops/TEST-DistElementwiseOpsTest-20221123040216.xml 2022-11-23T04:02:37.7265473Z 2022-11-23T04:02:37.7265882Z ##[endgroup] 2022-11-23T04:02:37.7266496Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_pointwise_ops (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_pointwise_ops_5h27fcew) 2022-11-23T04:02:37.7266839Z 2022-11-23T04:02:37.7267104Z Running distributed/_tensor/test_math_ops ... [2022-11-23 04:02:37.715541] 2022-11-23T04:02:37.7267789Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_math_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 04:02:37.716211] 2022-11-23T04:02:56.9829320Z 2022-11-23T04:02:56.9830227Z Expand the folded group to see the log file of distributed/_tensor/test_math_ops 2022-11-23T04:02:56.9832862Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_math_ops (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_math_ops_n_8m4zx8) 2022-11-23T04:02:56.9835100Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_math_ops 2022-11-23T04:02:56.9835817Z 2022-11-23T04:02:56.9836081Z Running tests... 2022-11-23T04:02:56.9837229Z ---------------------------------------------------------------------- 2022-11-23T04:02:56.9839241Z test_softmax_fwd (__main__.DistMathOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65393 2022-11-23T04:02:56.9840827Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65394 2022-11-23T04:02:56.9843672Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:56.9845429Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:56.9847633Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:56.9849352Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:56.9850693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:02:56.9851972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:02:56.9853706Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:56.9854863Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:56.9856420Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:56.9857652Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:56.9858753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:02:56.9860005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:02:56.9861746Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:56.9863563Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:56.9865054Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:56.9866374Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:56.9867194Z ok (5.608s) 2022-11-23T04:02:56.9868243Z test_softmax_with_bwd (__main__.DistMathOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65540 2022-11-23T04:02:56.9869550Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65541 2022-11-23T04:02:56.9871170Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:56.9872319Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:56.9873851Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:56.9875057Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:56.9876174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:02:56.9877389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:02:56.9879052Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:56.9880640Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:56.9882202Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:56.9883398Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:56.9884530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:02:56.9885761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:02:56.9887509Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:56.9889724Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:56.9891198Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:56.9892736Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:56.9893575Z ok (4.830s) 2022-11-23T04:02:56.9894598Z test_sum (__main__.DistMathOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65691 2022-11-23T04:02:56.9895625Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65692 2022-11-23T04:02:56.9896798Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:56.9897622Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:56.9898718Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:56.9899582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:56.9900386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:02:56.9901294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:02:56.9902485Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:02:56.9903308Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:02:56.9904418Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:02:56.9905266Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:02:56.9906078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:02:56.9906969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:02:56.9908216Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:56.9909527Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:02:56.9910603Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:56.9911547Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:02:56.9912143Z ok (4.530s) 2022-11-23T04:02:56.9912389Z 2022-11-23T04:02:56.9912910Z ---------------------------------------------------------------------- 2022-11-23T04:02:56.9913507Z Ran 3 tests in 14.971s 2022-11-23T04:02:56.9913794Z 2022-11-23T04:02:56.9913948Z OK 2022-11-23T04:02:56.9914186Z 2022-11-23T04:02:56.9914401Z Generating XML reports... 2022-11-23T04:02:56.9915509Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_math_ops/TEST-DistMathOpsTest-20221123040239.xml 2022-11-23T04:02:56.9916123Z 2022-11-23T04:02:56.9916698Z ##[endgroup] 2022-11-23T04:02:56.9917829Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_math_ops (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_math_ops_n_8m4zx8) 2022-11-23T04:02:56.9918594Z 2022-11-23T04:02:56.9919121Z Running distributed/_tensor/test_device_mesh ... [2022-11-23 04:02:56.983317] 2022-11-23T04:02:56.9920462Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_device_mesh.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 04:02:56.983988] 2022-11-23T04:04:34.3750264Z 2022-11-23T04:04:34.3755980Z Expand the folded group to see the log file of distributed/_tensor/test_device_mesh 2022-11-23T04:04:34.3759058Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_device_mesh (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_device_mesh_9_tn0ffl) 2022-11-23T04:04:34.3761983Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_device_mesh 2022-11-23T04:04:34.3763011Z 2022-11-23T04:04:34.3763383Z Running tests... 2022-11-23T04:04:34.3765606Z ---------------------------------------------------------------------- 2022-11-23T04:04:34.3768604Z test_all_gather_1d (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65905 2022-11-23T04:04:34.3770580Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65906 2022-11-23T04:04:34.3772203Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 65907 2022-11-23T04:04:34.3773897Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 65908 2022-11-23T04:04:34.3775648Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 65909 2022-11-23T04:04:34.3777287Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 65910 2022-11-23T04:04:34.3778975Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 65911 2022-11-23T04:04:34.3780914Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 65912 2022-11-23T04:04:34.3783775Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3785687Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3788706Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3790723Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3792528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T04:04:34.3795292Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3797136Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3799750Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3801594Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3803466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:04:34.3806209Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3808959Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3811793Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3813887Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3815814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:04:34.3818626Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3820176Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3822204Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3824318Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3825766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:04:34.3827986Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3829531Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3831441Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3832957Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3834346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T04:04:34.3836546Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3837961Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3839747Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3841194Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3842755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T04:04:34.3845035Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3846515Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3848447Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3849817Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3851300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:04:34.3853373Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3854761Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3856877Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3858236Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3859602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T04:04:34.3860477Z skip: Need at least 8 CUDA devices (5.155s) 2022-11-23T04:04:34.3861402Z test_all_gather_nd (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66441 2022-11-23T04:04:34.3862324Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66442 2022-11-23T04:04:34.3863199Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 66443 2022-11-23T04:04:34.3863935Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 66444 2022-11-23T04:04:34.3864693Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 66445 2022-11-23T04:04:34.3865342Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 66446 2022-11-23T04:04:34.3865988Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 66447 2022-11-23T04:04:34.3866591Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 66448 2022-11-23T04:04:34.3867473Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3868010Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3868603Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3869194Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3869624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:04:34.3870240Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3870682Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3871241Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3871699Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3872123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:04:34.3872734Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3873239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3873822Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3874277Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3874685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:04:34.3875299Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3875736Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3876309Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3876760Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3877187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T04:04:34.3877801Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3878234Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3878796Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3879244Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3879667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T04:04:34.3880284Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3880713Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3881297Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3881756Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3882161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T04:04:34.3882773Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3883207Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3883781Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3884238Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3884662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:04:34.3885275Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3885699Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3886344Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3886793Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3887207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T04:04:34.3887590Z skip: Need at least 8 CUDA devices (4.589s) 2022-11-23T04:04:34.3888185Z test_all_gather_uneven (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66977 2022-11-23T04:04:34.3888747Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66978 2022-11-23T04:04:34.3889268Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 66979 2022-11-23T04:04:34.3889773Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 66980 2022-11-23T04:04:34.3890370Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 66981 2022-11-23T04:04:34.3890891Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 66982 2022-11-23T04:04:34.3891422Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 66983 2022-11-23T04:04:34.3891953Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 66984 2022-11-23T04:04:34.3892708Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3893232Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3893909Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3894448Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3894956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T04:04:34.3895703Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3896223Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3896910Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3897452Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3897945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:04:34.3898674Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3899195Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3899858Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3900311Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3900742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:04:34.3901350Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3901771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3902339Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3902790Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3903213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T04:04:34.3903824Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3904256Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3904905Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3906316Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3906726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T04:04:34.3907336Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3907764Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3908332Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3908778Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3909201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T04:04:34.3909869Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3910291Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3910867Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3911320Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3911738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:04:34.3912343Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3912778Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3913351Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3913790Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3914218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:04:34.3914599Z skip: Need at least 8 CUDA devices (4.582s) 2022-11-23T04:04:34.3915060Z test_all_reduce_1d (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67513 2022-11-23T04:04:34.3915570Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67514 2022-11-23T04:04:34.3916005Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 67515 2022-11-23T04:04:34.3916439Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 67516 2022-11-23T04:04:34.3916865Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 67517 2022-11-23T04:04:34.3917300Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 67518 2022-11-23T04:04:34.3917731Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 67519 2022-11-23T04:04:34.3918176Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 67520 2022-11-23T04:04:34.3918782Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3919214Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3919783Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3920233Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3920643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:04:34.3921252Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3921682Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3922260Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3922771Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3923190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T04:04:34.3923807Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3924226Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3924805Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3925252Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3925673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:04:34.3926343Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3926783Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3927363Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3927954Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3928390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T04:04:34.3929108Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3929620Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3930309Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3930846Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3931360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T04:04:34.3932105Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3932611Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3933305Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3933847Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3934353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T04:04:34.3935084Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3935607Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3936296Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3936821Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3937329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:04:34.3938060Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3938579Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3939264Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3939805Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3940228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:04:34.3940598Z skip: Need at least 8 CUDA devices (4.596s) 2022-11-23T04:04:34.3941060Z test_all_reduce_nd (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68049 2022-11-23T04:04:34.3941664Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68050 2022-11-23T04:04:34.3942102Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 68051 2022-11-23T04:04:34.3942540Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 68052 2022-11-23T04:04:34.3942976Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 68053 2022-11-23T04:04:34.3943411Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 68054 2022-11-23T04:04:34.3943834Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 68055 2022-11-23T04:04:34.3944268Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 68056 2022-11-23T04:04:34.3944879Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3945368Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3945950Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3946404Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3946827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:04:34.3947428Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3947865Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3948441Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3948900Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3949327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:04:34.3949943Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3950376Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3950953Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3951397Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3951823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T04:04:34.3952433Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3952865Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3953440Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3953904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3954325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:04:34.3954928Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3955361Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3955933Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3956384Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3956802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T04:04:34.3957417Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3957853Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3958584Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3959127Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3959628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:04:34.3960359Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3960885Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3961460Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3961913Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3962319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T04:04:34.3963004Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3963443Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3964018Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3964478Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3964901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T04:04:34.3965281Z skip: Need at least 8 CUDA devices (4.683s) 2022-11-23T04:04:34.3965739Z test_all_to_all_1d (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68585 2022-11-23T04:04:34.3966238Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68586 2022-11-23T04:04:34.3966682Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 68587 2022-11-23T04:04:34.3967123Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 68588 2022-11-23T04:04:34.3967560Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 68589 2022-11-23T04:04:34.3968291Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 68590 2022-11-23T04:04:34.3968863Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 68591 2022-11-23T04:04:34.3969390Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 68592 2022-11-23T04:04:34.3970131Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3970646Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3971333Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3971875Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3972395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:04:34.3973138Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3973661Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3974340Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3974887Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3975397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T04:04:34.3976124Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3976642Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3977438Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3977977Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3978462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T04:04:34.3979198Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3979715Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3980374Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3980824Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3981243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T04:04:34.3981907Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3982354Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3982922Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3983374Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3983796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T04:04:34.3984401Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3984833Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3985400Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3985850Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3986266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:04:34.3986879Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3987311Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3987886Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3988336Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3988758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:04:34.3989366Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3989787Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3990365Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3990822Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3991246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:04:34.3991629Z skip: Need at least 8 CUDA devices (4.673s) 2022-11-23T04:04:34.3992081Z test_all_to_all_nd (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69121 2022-11-23T04:04:34.3992593Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69122 2022-11-23T04:04:34.3993023Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 69123 2022-11-23T04:04:34.3993457Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 69124 2022-11-23T04:04:34.3993889Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 69125 2022-11-23T04:04:34.3994389Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 69126 2022-11-23T04:04:34.3994825Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 69127 2022-11-23T04:04:34.3995256Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 69128 2022-11-23T04:04:34.3995871Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3996291Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3996865Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3997318Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.3997737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:04:34.3998399Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.3998836Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.3999414Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.3999870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4000282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:04:34.4000895Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4001325Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4001896Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4002352Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4002781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T04:04:34.4003434Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4003939Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4004623Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4005166Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4005673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T04:04:34.4006409Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4006931Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4007622Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4008296Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4008806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:04:34.4009545Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4010066Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4010757Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4011311Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4011816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T04:04:34.4012550Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4013163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4013856Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4014402Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4014910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:04:34.4015641Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4016163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4016847Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4017376Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4017942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T04:04:34.4018406Z skip: Need at least 8 CUDA devices (4.674s) 2022-11-23T04:04:34.4018957Z test_broadcast_1d (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69657 2022-11-23T04:04:34.4019575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69658 2022-11-23T04:04:34.4020111Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 69659 2022-11-23T04:04:34.4020557Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 69660 2022-11-23T04:04:34.4020984Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 69661 2022-11-23T04:04:34.4021420Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 69662 2022-11-23T04:04:34.4021858Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 69663 2022-11-23T04:04:34.4022289Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 69664 2022-11-23T04:04:34.4022910Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4023349Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4023927Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4024367Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4024796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:04:34.4025405Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4025842Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4026419Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4026877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4027299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:04:34.4027900Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4028333Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4028907Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4029359Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4029780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T04:04:34.4030393Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4030830Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4031476Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4031920Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4032346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:04:34.4032956Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4033386Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4033961Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4034409Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4034836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T04:04:34.4035512Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4035956Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4036534Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4036987Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4037405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T04:04:34.4038011Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4038444Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4039009Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4039465Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4039892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:04:34.4040502Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4040936Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4041508Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4041960Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4042368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T04:04:34.4042753Z skip: Need at least 8 CUDA devices (4.673s) 2022-11-23T04:04:34.4043214Z test_broadcast_nd (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70193 2022-11-23T04:04:34.4043742Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70194 2022-11-23T04:04:34.4044180Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 70195 2022-11-23T04:04:34.4044614Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 70196 2022-11-23T04:04:34.4045048Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 70197 2022-11-23T04:04:34.4045472Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 70198 2022-11-23T04:04:34.4045906Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 70199 2022-11-23T04:04:34.4046341Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 70200 2022-11-23T04:04:34.4046950Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4047384Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4048275Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4049069Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4049577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:04:34.4050315Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4050829Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4051514Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4052053Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4052555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:04:34.4053379Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4053914Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4054597Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4055145Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4055650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T04:04:34.4056382Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4056904Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4057594Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4058143Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4058639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T04:04:34.4059372Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4059898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4060518Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4060967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4061388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T04:04:34.4062001Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4062438Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4063006Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4063459Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4063882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:04:34.4064493Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4064923Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4065498Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4065954Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4066364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T04:04:34.4066981Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4067477Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4068056Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4068508Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4068932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:04:34.4069315Z skip: Need at least 8 CUDA devices (4.680s) 2022-11-23T04:04:34.4069769Z test_reduce_scatter_1d (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70729 2022-11-23T04:04:34.4070294Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70730 2022-11-23T04:04:34.4070731Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 70731 2022-11-23T04:04:34.4071219Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 70732 2022-11-23T04:04:34.4071659Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 70733 2022-11-23T04:04:34.4072095Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 70734 2022-11-23T04:04:34.4072530Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 70735 2022-11-23T04:04:34.4072950Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 70736 2022-11-23T04:04:34.4073563Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4073998Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4074572Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4075023Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4075449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:04:34.4076061Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4076479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4077056Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4077514Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4077930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T04:04:34.4078542Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4078972Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4079552Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4080013Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4080429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:04:34.4081037Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4081467Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4082034Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4082484Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4082903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:04:34.4083517Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4084012Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4084587Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4085044Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4085465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:04:34.4086074Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4086508Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4087083Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4087522Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4087996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T04:04:34.4088682Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4089209Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4089904Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4090448Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4090954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T04:04:34.4091688Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4092190Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4092880Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4093443Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4093947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T04:04:34.4094403Z skip: Need at least 8 CUDA devices (4.574s) 2022-11-23T04:04:34.4094967Z test_reduce_scatter_nd (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71265 2022-11-23T04:04:34.4095595Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71266 2022-11-23T04:04:34.4096108Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 71267 2022-11-23T04:04:34.4096622Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 71268 2022-11-23T04:04:34.4097140Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 71269 2022-11-23T04:04:34.4097665Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 71270 2022-11-23T04:04:34.4098199Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 71271 2022-11-23T04:04:34.4098719Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 71272 2022-11-23T04:04:34.4099459Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4099971Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4100596Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4101055Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4101475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:04:34.4102085Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4102521Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4103189Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4103626Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4104052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T04:04:34.4104664Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4105096Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4105677Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4106126Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4106545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T04:04:34.4107207Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4107642Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4108225Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4108681Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4109104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T04:04:34.4109720Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4110154Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4110724Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4111169Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4111591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T04:04:34.4112206Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4112637Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4113209Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4113664Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4114082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:04:34.4114681Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4115110Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4115691Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4116140Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4116558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:04:34.4117166Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4117598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4118163Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4118619Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4119042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:04:34.4119419Z skip: Need at least 8 CUDA devices (4.786s) 2022-11-23T04:04:34.4119962Z test_reduce_scatter_uneven (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71801 2022-11-23T04:04:34.4120488Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71802 2022-11-23T04:04:34.4120928Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 71803 2022-11-23T04:04:34.4121352Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 71804 2022-11-23T04:04:34.4121790Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 71805 2022-11-23T04:04:34.4122227Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 71806 2022-11-23T04:04:34.4122670Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 71807 2022-11-23T04:04:34.4123101Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 71808 2022-11-23T04:04:34.4123841Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4124284Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4124851Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4125306Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4125727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T04:04:34.4126340Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4126771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4127347Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4127901Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4128342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:04:34.4129000Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4129511Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4130193Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4130737Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4131247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:04:34.4131975Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4132492Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4133175Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4133725Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4134231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T04:04:34.4134969Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4135485Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4136178Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4136717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4137207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T04:04:34.4137949Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4138564Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4139259Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4139801Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4140308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T04:04:34.4140926Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4141357Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4141919Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4142373Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4142863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:04:34.4143485Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4143920Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4144493Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4144945Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4145357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:04:34.4145730Z skip: Need at least 8 CUDA devices (4.775s) 2022-11-23T04:04:34.4146194Z test_scatter_1d (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72337 2022-11-23T04:04:34.4146707Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72338 2022-11-23T04:04:34.4147155Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 72339 2022-11-23T04:04:34.4147593Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 72340 2022-11-23T04:04:34.4148032Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 72341 2022-11-23T04:04:34.4148459Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 72342 2022-11-23T04:04:34.4148897Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 72343 2022-11-23T04:04:34.4149331Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 72344 2022-11-23T04:04:34.4149937Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4150371Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4150949Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4151403Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4151811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:04:34.4152427Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4152856Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4153435Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4153884Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4154316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:04:34.4154932Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4155428Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4156007Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4156459Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4156880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T04:04:34.4157494Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4157931Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4158509Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4158962Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4159372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T04:04:34.4160039Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4160473Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4161050Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4161500Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4161927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:04:34.4162537Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4162956Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4163535Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4163995Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4164421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T04:04:34.4165038Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4165471Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4166049Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4166492Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4166916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:04:34.4167524Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4168001Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4168581Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4169102Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4169608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T04:04:34.4170056Z skip: Need at least 8 CUDA devices (4.589s) 2022-11-23T04:04:34.4170612Z test_scatter_nd (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72873 2022-11-23T04:04:34.4171231Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72874 2022-11-23T04:04:34.4171757Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 72875 2022-11-23T04:04:34.4172285Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 72876 2022-11-23T04:04:34.4172821Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 72877 2022-11-23T04:04:34.4173428Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 72878 2022-11-23T04:04:34.4173944Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 72879 2022-11-23T04:04:34.4174454Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 72880 2022-11-23T04:04:34.4175195Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4175717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4176405Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4176944Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4177452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T04:04:34.4178255Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4178767Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4179467Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4180021Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4180484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T04:04:34.4181095Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4181528Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4182105Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4182551Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4182981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:04:34.4183592Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4184032Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4184607Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4185054Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4185478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:04:34.4186077Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4186516Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4187097Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4187546Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4187967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T04:04:34.4188578Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4189013Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4189584Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4190024Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4190445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:04:34.4191057Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4191564Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4192147Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4192602Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4193028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:04:34.4193627Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4194064Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4194634Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4195091Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4195563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T04:04:34.4195949Z skip: Need at least 8 CUDA devices (4.676s) 2022-11-23T04:04:34.4196415Z test_scatter_uneven (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73409 2022-11-23T04:04:34.4196926Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73410 2022-11-23T04:04:34.4197365Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 73411 2022-11-23T04:04:34.4197801Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 73412 2022-11-23T04:04:34.4198240Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 73413 2022-11-23T04:04:34.4198673Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 73414 2022-11-23T04:04:34.4199108Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 73415 2022-11-23T04:04:34.4199553Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 73416 2022-11-23T04:04:34.4200153Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4200598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4201174Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4201627Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4202052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T04:04:34.4202660Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4203093Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4203662Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4204123Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4204549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:04:34.4205160Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4205592Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4206167Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4206621Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4207047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:04:34.4207649Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4208291Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4208922Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4209467Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4209971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T04:04:34.4210704Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4211219Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4211889Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4212435Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4213009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T04:04:34.4213765Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4214285Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4214978Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4215518Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4216007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T04:04:34.4216755Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4217274Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4217971Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4218522Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4219029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:04:34.4219759Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4220284Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4220885Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4221336Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4221824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:04:34.4222280Z skip: Need at least 8 CUDA devices (5.879s) 2022-11-23T04:04:34.4222820Z test_device_mesh_2d (__main__.DeviceMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73945 2022-11-23T04:04:34.4223425Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73946 2022-11-23T04:04:34.4223947Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 73947 2022-11-23T04:04:34.4224459Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 73948 2022-11-23T04:04:34.4225006Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 73949 2022-11-23T04:04:34.4225557Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 73950 2022-11-23T04:04:34.4226086Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 73951 2022-11-23T04:04:34.4226621Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 73952 2022-11-23T04:04:34.4227361Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4227904Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4228687Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4229271Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4229791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:04:34.4230541Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4231059Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4231751Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4232299Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4232795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:04:34.4233627Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4234178Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4234892Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4235447Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4235961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T04:04:34.4236722Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4237239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4237912Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4238506Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4239017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:04:34.4239751Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4240268Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4240869Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4241345Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4241757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T04:04:34.4242368Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4242805Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4243390Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4243842Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4244267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T04:04:34.4244887Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4245327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4245890Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4246342Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4246764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T04:04:34.4247384Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4248058Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4248652Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4249153Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4249570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:04:34.4249973Z skip: Need at least 8 CUDA devices (5.678s) 2022-11-23T04:04:34.4250442Z test_device_mesh_2d_from_dim_groups (__main__.DeviceMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74481 2022-11-23T04:04:34.4250967Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74482 2022-11-23T04:04:34.4251407Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 74483 2022-11-23T04:04:34.4251919Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 74484 2022-11-23T04:04:34.4252366Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 74485 2022-11-23T04:04:34.4252803Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 74486 2022-11-23T04:04:34.4253232Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 74487 2022-11-23T04:04:34.4253676Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 74488 2022-11-23T04:04:34.4254291Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4254754Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4255328Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4255806Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4256249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:04:34.4256860Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4257314Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4257911Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4258363Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4258808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T04:04:34.4259432Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4259867Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4260446Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4260909Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4261332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:04:34.4261937Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4262372Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4262953Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4263408Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4263830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T04:04:34.4264451Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4264963Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4265534Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4265991Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4266422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T04:04:34.4267037Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4267471Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4268048Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4268509Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4268979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:04:34.4269605Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4270041Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4270616Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4271069Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4271496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:04:34.4272114Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4272548Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4273117Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4273574Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4273999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T04:04:34.4274381Z skip: Need at least 8 CUDA devices (5.177s) 2022-11-23T04:04:34.4274847Z test_device_mesh_dim_groups_error (__main__.DeviceMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75017 2022-11-23T04:04:34.4275364Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75018 2022-11-23T04:04:34.4275814Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 75019 2022-11-23T04:04:34.4276242Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 75020 2022-11-23T04:04:34.4276677Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 75021 2022-11-23T04:04:34.4277124Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 75022 2022-11-23T04:04:34.4277578Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 75023 2022-11-23T04:04:34.4278029Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 75024 2022-11-23T04:04:34.4278640Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4279077Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4279641Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4280100Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4280535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:04:34.4281153Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4281678Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4282404Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4282866Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4283277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:04:34.4283897Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4284331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4284905Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4285359Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4285853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:04:34.4286499Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4286941Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4287530Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4288107Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4288533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T04:04:34.4289193Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4289631Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4290207Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4290674Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4291103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T04:04:34.4291724Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4292146Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4292720Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4293178Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4293607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T04:04:34.4294219Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4294665Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4295248Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4295687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4296117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T04:04:34.4296729Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4297168Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4297749Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4298208Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4298634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:04:34.4299095Z skip: Need at least 8 CUDA devices (5.178s) 2022-11-23T04:04:34.4299545Z test_device_mesh_nd (__main__.DeviceMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75553 2022-11-23T04:04:34.4300049Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75554 2022-11-23T04:04:34.4300491Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 75555 2022-11-23T04:04:34.4300929Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 75556 2022-11-23T04:04:34.4301372Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 75557 2022-11-23T04:04:34.4301811Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 75558 2022-11-23T04:04:34.4302237Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 75559 2022-11-23T04:04:34.4302681Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 75560 2022-11-23T04:04:34.4303357Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4303805Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4304401Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4304864Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4305295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T04:04:34.4305939Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4306358Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4306942Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4307407Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4307836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:04:34.4308468Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4308913Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4309644Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4310098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4310536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:04:34.4311170Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4311638Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4312221Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4312676Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4313105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:04:34.4313718Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4314151Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4314715Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4315172Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4315604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T04:04:34.4316227Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4316741Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4317324Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4317782Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4318206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T04:04:34.4318807Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4319244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4319828Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4320283Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4320763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:04:34.4321381Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:04:34.4321818Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:04:34.4322383Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:04:34.4322837Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:04:34.4323267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T04:04:34.4323651Z skip: Need at least 8 CUDA devices (5.700s) 2022-11-23T04:04:34.4323834Z 2022-11-23T04:04:34.4324117Z ---------------------------------------------------------------------- 2022-11-23T04:04:34.4324444Z Ran 19 tests in 93.321s 2022-11-23T04:04:34.4324605Z 2022-11-23T04:04:34.4324710Z OK (skipped=19) 2022-11-23T04:04:34.4324859Z 2022-11-23T04:04:34.4324964Z Generating XML reports... 2022-11-23T04:04:34.4325609Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_device_mesh/TEST-DeviceMeshCollectiveTest-20221123040258.xml 2022-11-23T04:04:34.4326388Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_device_mesh/TEST-DeviceMeshTest-20221123040258.xml 2022-11-23T04:04:34.4326732Z 2022-11-23T04:04:34.4327243Z ##[endgroup] 2022-11-23T04:04:34.4327905Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_device_mesh (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_device_mesh_9_tn0ffl) 2022-11-23T04:04:34.4328248Z 2022-11-23T04:04:34.4328516Z Running distributed/_tensor/test_api ... [2022-11-23 04:04:34.379267] 2022-11-23T04:04:34.4329189Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_api.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 04:04:34.380110] 2022-11-23T04:05:00.5239967Z 2022-11-23T04:05:00.5240871Z Expand the folded group to see the log file of distributed/_tensor/test_api 2022-11-23T04:05:00.5242998Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_api (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_api_srkcikt1) 2022-11-23T04:05:00.5249269Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_api 2022-11-23T04:05:00.5250126Z 2022-11-23T04:05:00.5255167Z Running tests... 2022-11-23T04:05:00.5256635Z ---------------------------------------------------------------------- 2022-11-23T04:05:00.5258003Z test_distribute_module (__main__.DTensorAPITest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76156 2022-11-23T04:05:00.5259356Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76157 2022-11-23T04:05:00.5260756Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 76158 2022-11-23T04:05:00.5262825Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 76159 2022-11-23T04:05:00.5264754Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:00.5266241Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:00.5268067Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:00.5269574Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:00.5271189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:05:00.5273406Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:00.5274798Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:00.5277064Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:00.5278305Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:00.5279417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:05:00.5281094Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:00.5282227Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:00.5283751Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:00.5284944Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:00.5286057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:05:00.5287683Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:00.5288993Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:00.5290527Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:00.5291706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:00.5292816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:05:00.5293825Z skip: Need at least 4 CUDA devices (4.732s) 2022-11-23T04:05:00.5295056Z test_distribute_module_input_fn_output_fn (__main__.DTensorAPITest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76424 2022-11-23T04:05:00.5296555Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76425 2022-11-23T04:05:00.5297700Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 76426 2022-11-23T04:05:00.5298854Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 76427 2022-11-23T04:05:00.5300486Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:00.5301626Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:00.5303137Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:00.5304304Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:00.5305410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:05:00.5307038Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:00.5308172Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:00.5309692Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:00.5311073Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:00.5312181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:05:00.5313809Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:00.5314944Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:00.5316465Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:00.5317644Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:00.5318755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:05:00.5320375Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:00.5321660Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:00.5323187Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:00.5324371Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:00.5325477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:05:00.5326480Z skip: Need at least 4 CUDA devices (4.249s) 2022-11-23T04:05:00.5327655Z test_distribute_tensor (__main__.DTensorAPITest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76692 2022-11-23T04:05:00.5329350Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76693 2022-11-23T04:05:00.5330502Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 76694 2022-11-23T04:05:00.5331633Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 76695 2022-11-23T04:05:00.5333279Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:00.5334428Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:00.5335940Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:00.5337125Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:00.5338239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:05:00.5339868Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:00.5340546Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:00.5341182Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:00.5341633Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:00.5342069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:05:00.5342683Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:00.5343123Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:00.5343697Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:00.5344154Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:00.5344566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:05:00.5345177Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:00.5345611Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:00.5346189Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:00.5346823Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:00.5347254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:05:00.5347639Z skip: Need at least 4 CUDA devices (4.148s) 2022-11-23T04:05:00.5348083Z test_distribute_tensor_errors (__main__.DTensorAPITest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76960 2022-11-23T04:05:00.5348591Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76961 2022-11-23T04:05:00.5349024Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 76962 2022-11-23T04:05:00.5349461Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 76963 2022-11-23T04:05:00.5350071Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:00.5350565Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:00.5351148Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:00.5351593Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:00.5352021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:05:00.5352634Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:00.5353064Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:00.5353639Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:00.5354090Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:00.5354522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:05:00.5355142Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:00.5355567Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:00.5356149Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:00.5356601Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:00.5357027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:05:00.5357640Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:00.5358074Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:00.5358656Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:00.5359102Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:00.5359524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:05:00.5359908Z skip: Need at least 4 CUDA devices (4.147s) 2022-11-23T04:05:00.5360375Z test_distribute_tensor_uneven_sharding (__main__.DTensorAPITest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77228 2022-11-23T04:05:00.5360893Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77229 2022-11-23T04:05:00.5361331Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 77230 2022-11-23T04:05:00.5361763Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 77231 2022-11-23T04:05:00.5362358Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:00.5362795Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:00.5363437Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:00.5363893Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:00.5364316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:05:00.5364929Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:00.5365365Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:00.5365926Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:00.5366377Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:00.5366800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:05:00.5367469Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:00.5367945Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:00.5368562Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:00.5369095Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:00.5369603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:05:00.5370327Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:00.5370852Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:00.5371547Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:00.5372098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:00.5372607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:05:00.5373068Z skip: Need at least 4 CUDA devices (4.449s) 2022-11-23T04:05:00.5373285Z 2022-11-23T04:05:00.5373613Z ---------------------------------------------------------------------- 2022-11-23T04:05:00.5373982Z Ran 5 tests in 21.728s 2022-11-23T04:05:00.5374165Z 2022-11-23T04:05:00.5374286Z OK (skipped=5) 2022-11-23T04:05:00.5374453Z 2022-11-23T04:05:00.5374597Z Generating XML reports... 2022-11-23T04:05:00.5375276Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_api/TEST-DTensorAPITest-20221123040436.xml 2022-11-23T04:05:00.5375648Z 2022-11-23T04:05:00.5376213Z ##[endgroup] 2022-11-23T04:05:00.5376912Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_api (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_api_srkcikt1) 2022-11-23T04:05:00.5377297Z 2022-11-23T04:05:00.5377654Z Running distributed/_tensor/parallel/test_tp_style ... [2022-11-23 04:05:00.525057] 2022-11-23T04:05:00.5378515Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/parallel/test_tp_style.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 04:05:00.525723] 2022-11-23T04:05:40.2351063Z 2022-11-23T04:05:40.2352814Z Expand the folded group to see the log file of distributed/_tensor/parallel/test_tp_style 2022-11-23T04:05:40.2355153Z ##[group]PRINTING LOG FILE of distributed/_tensor/parallel/test_tp_style (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-parallel-test_tp_style_5mpnlmqe) 2022-11-23T04:05:40.2358071Z Test results will be stored in test-reports/python-unittest/distributed._tensor.parallel.test_tp_style 2022-11-23T04:05:40.2358883Z 2022-11-23T04:05:40.2359157Z Running tests... 2022-11-23T04:05:40.2360489Z ---------------------------------------------------------------------- 2022-11-23T04:05:40.2362381Z test_colwise_parallel_style (__main__.TensorParallelStyleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77563 2022-11-23T04:05:40.2364589Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77564 2022-11-23T04:05:40.2365769Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 77565 2022-11-23T04:05:40.2374142Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 77566 2022-11-23T04:05:40.2376212Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:40.2377385Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:40.2379025Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:40.2380239Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:40.2381894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:05:40.2383602Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:40.2384743Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:40.2386288Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:40.2387480Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:40.2388577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:05:40.2390212Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:40.2391344Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:40.2392866Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:40.2394072Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:40.2395184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:05:40.2396809Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:40.2397948Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:40.2399452Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:40.2400629Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:40.2401724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:05:40.2402720Z skip: Need at least 4 CUDA devices (4.837s) 2022-11-23T04:05:40.2403900Z test_make_input_replicate_1d (__main__.TensorParallelStyleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77831 2022-11-23T04:05:40.2405067Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77832 2022-11-23T04:05:40.2406028Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 77833 2022-11-23T04:05:40.2406979Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 77834 2022-11-23T04:05:40.2408509Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:40.2409647Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:40.2411180Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:40.2412354Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:40.2413448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:05:40.2415332Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:40.2416444Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:40.2417964Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:40.2419134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:40.2420238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:05:40.2421852Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:40.2422972Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:40.2424299Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:40.2425412Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:40.2426359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:05:40.2427742Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:40.2428684Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:40.2429954Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:40.2430950Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:40.2431881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:05:40.2432708Z skip: Need at least 4 CUDA devices (4.446s) 2022-11-23T04:05:40.2433734Z test_make_input_shard_1d (__main__.TensorParallelStyleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78099 2022-11-23T04:05:40.2434893Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78100 2022-11-23T04:05:40.2435855Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 78101 2022-11-23T04:05:40.2436809Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 78102 2022-11-23T04:05:40.2438149Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:40.2439106Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:40.2440384Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:40.2441371Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:40.2442300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:05:40.2443676Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:40.2444632Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:40.2445916Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:40.2447022Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:40.2448214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:05:40.2449799Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:40.2450928Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:40.2452435Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:40.2453618Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:40.2454899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:05:40.2456533Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:40.2457658Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:40.2459178Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:40.2460356Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:40.2461461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:05:40.2462459Z skip: Need at least 4 CUDA devices (4.046s) 2022-11-23T04:05:40.2463524Z test_make_output_replicate_1d (__main__.TensorParallelStyleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78367 2022-11-23T04:05:40.2464177Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78368 2022-11-23T04:05:40.2464613Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 78369 2022-11-23T04:05:40.2465039Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 78370 2022-11-23T04:05:40.2465642Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:40.2466069Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:40.2466635Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:40.2467085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:40.2467492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:05:40.2468096Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:40.2468533Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:40.2469102Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:40.2469546Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:40.2469963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:05:40.2470570Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:40.2470989Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:40.2471557Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:40.2472001Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:40.2472416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:05:40.2473025Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:40.2473453Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:40.2474021Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:40.2474460Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:40.2474877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:05:40.2475250Z skip: Need at least 4 CUDA devices (4.852s) 2022-11-23T04:05:40.2475710Z test_make_output_shard_1d (__main__.TensorParallelStyleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78635 2022-11-23T04:05:40.2476225Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78636 2022-11-23T04:05:40.2476720Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 78637 2022-11-23T04:05:40.2477147Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 78638 2022-11-23T04:05:40.2477745Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:40.2478170Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:40.2478739Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:40.2479184Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:40.2479602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:05:40.2480208Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:40.2480631Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:40.2481342Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:40.2481787Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:40.2482205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:05:40.2482820Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:40.2483246Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:40.2483815Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:40.2484258Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:40.2484672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:05:40.2485278Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:40.2485707Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:40.2486277Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:40.2486722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:40.2487137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:05:40.2487512Z skip: Need at least 4 CUDA devices (4.246s) 2022-11-23T04:05:40.2488033Z test_make_output_tensor (__main__.TensorParallelStyleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78903 2022-11-23T04:05:40.2488570Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78904 2022-11-23T04:05:40.2489079Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 78905 2022-11-23T04:05:40.2489591Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 78906 2022-11-23T04:05:40.2490305Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:40.2490814Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:40.2491493Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:40.2492025Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:40.2492513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:05:40.2493244Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:40.2493754Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:40.2494439Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:40.2495120Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:40.2495625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:05:40.2496364Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:40.2496876Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:40.2497553Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:40.2498086Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:40.2498587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:05:40.2499386Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:40.2499909Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:40.2500597Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:40.2501130Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:40.2501625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:05:40.2502077Z skip: Need at least 4 CUDA devices (4.146s) 2022-11-23T04:05:40.2502631Z test_prepare_output_error (__main__.TensorParallelStyleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79171 2022-11-23T04:05:40.2503251Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79172 2022-11-23T04:05:40.2503775Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 79173 2022-11-23T04:05:40.2504241Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 79174 2022-11-23T04:05:40.2504844Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:40.2505264Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:40.2505832Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:40.2506279Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:40.2506699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:05:40.2507314Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:40.2507739Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:40.2508314Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:40.2508755Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:40.2509175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:05:40.2509785Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:40.2510209Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:40.2510778Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:40.2511224Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:40.2511642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:05:40.2512248Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:40.2512729Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:40.2513304Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:40.2513749Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:40.2514166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:05:40.2514547Z skip: Need at least 4 CUDA devices (4.344s) 2022-11-23T04:05:40.2515024Z test_rowwise_parallel_style (__main__.TensorParallelStyleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79439 2022-11-23T04:05:40.2515545Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79440 2022-11-23T04:05:40.2515969Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 79441 2022-11-23T04:05:40.2516402Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 79442 2022-11-23T04:05:40.2517065Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:40.2517504Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:40.2518083Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:40.2518535Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:40.2518957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:05:40.2519554Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:40.2519989Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:40.2520565Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:40.2521027Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:40.2521461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:05:40.2522076Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:40.2522510Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:40.2523074Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:40.2523527Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:40.2523952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:05:40.2524563Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:05:40.2524994Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:05:40.2525581Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:05:40.2526033Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:05:40.2526457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:05:40.2526828Z skip: Need at least 4 CUDA devices (4.648s) 2022-11-23T04:05:40.2527008Z 2022-11-23T04:05:40.2527285Z ---------------------------------------------------------------------- 2022-11-23T04:05:40.2527606Z Ran 8 tests in 35.568s 2022-11-23T04:05:40.2527816Z 2022-11-23T04:05:40.2527914Z OK (skipped=8) 2022-11-23T04:05:40.2528060Z 2022-11-23T04:05:40.2528177Z Generating XML reports... 2022-11-23T04:05:40.2528909Z Generated XML report: test-reports/python-unittest/distributed._tensor.parallel.test_tp_style/TEST-TensorParallelStyleTest-20221123040502.xml 2022-11-23T04:05:40.2529328Z 2022-11-23T04:05:40.2529918Z ##[endgroup] 2022-11-23T04:05:40.2530779Z FINISHED PRINTING LOG FILE of distributed/_tensor/parallel/test_tp_style (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-parallel-test_tp_style_5mpnlmqe) 2022-11-23T04:05:40.2531205Z 2022-11-23T04:05:40.2531573Z Running distributed/_tensor/parallel/test_parallelize_api ... [2022-11-23 04:05:40.236214] 2022-11-23T04:05:40.2532466Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/parallel/test_parallelize_api.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 04:05:40.237031] 2022-11-23T04:06:02.5753876Z 2022-11-23T04:06:02.5755019Z Expand the folded group to see the log file of distributed/_tensor/parallel/test_parallelize_api 2022-11-23T04:06:02.5757895Z ##[group]PRINTING LOG FILE of distributed/_tensor/parallel/test_parallelize_api (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-parallel-test_parallelize_api_31nqe95i) 2022-11-23T04:06:02.5766129Z Test results will be stored in test-reports/python-unittest/distributed._tensor.parallel.test_parallelize_api 2022-11-23T04:06:02.5767242Z 2022-11-23T04:06:02.5767562Z Running tests... 2022-11-23T04:06:02.5769145Z ---------------------------------------------------------------------- 2022-11-23T04:06:02.5770608Z test_creat_1d_device_mesh (__main__.TensorParallelAPITests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79774 2022-11-23T04:06:02.5772346Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79775 2022-11-23T04:06:02.5773518Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 79776 2022-11-23T04:06:02.5774696Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 79777 2022-11-23T04:06:02.5776911Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:06:02.5778117Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:06:02.5779750Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:06:02.5780962Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:06:02.5782307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:06:02.5784575Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:06:02.5786014Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:06:02.5788173Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:06:02.5789713Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:06:02.5791094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:06:02.5792959Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:06:02.5794235Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:06:02.5795915Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:06:02.5797240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:06:02.5798466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:06:02.5800282Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:06:02.5801532Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:06:02.5803213Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:06:02.5804539Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:06:02.5806120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:06:02.5807242Z skip: Need at least 4 CUDA devices (4.917s) 2022-11-23T04:06:02.5808693Z test_creat_1d_device_mesh_error (__main__.TensorParallelAPITests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80042 2022-11-23T04:06:02.5810086Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80043 2022-11-23T04:06:02.5811240Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 80044 2022-11-23T04:06:02.5812385Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 80045 2022-11-23T04:06:02.5814044Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:06:02.5815193Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:06:02.5816889Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:06:02.5818134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:06:02.5819248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:06:02.5820906Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:06:02.5822054Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:06:02.5823579Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:06:02.5824769Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:06:02.5825867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:06:02.5827512Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:06:02.5832470Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:06:02.5834099Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:06:02.5835295Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:06:02.5836411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:06:02.5838045Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:06:02.5839182Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:06:02.5840726Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:06:02.5841914Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:06:02.5843038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:06:02.5844069Z skip: Need at least 4 CUDA devices (4.142s) 2022-11-23T04:06:02.5845288Z test_parallelize_mlp (__main__.TensorParallelAPITests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80310 2022-11-23T04:06:02.5846657Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80311 2022-11-23T04:06:02.5848134Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 80312 2022-11-23T04:06:02.5849429Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 80313 2022-11-23T04:06:02.5851085Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:06:02.5852241Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:06:02.5853784Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:06:02.5854989Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:06:02.5856429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:06:02.5858071Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:06:02.5858842Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:06:02.5859483Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:06:02.5859936Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:06:02.5860359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:06:02.5860974Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:06:02.5861407Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:06:02.5862180Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:06:02.5862639Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:06:02.5863050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:06:02.5863669Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:06:02.5864099Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:06:02.5864673Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:06:02.5865124Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:06:02.5865546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:06:02.5865929Z skip: Need at least 4 CUDA devices (4.254s) 2022-11-23T04:06:02.5866405Z test_parallelize_mlp_error (__main__.TensorParallelAPITests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80578 2022-11-23T04:06:02.5866933Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80579 2022-11-23T04:06:02.5867370Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 80580 2022-11-23T04:06:02.5867803Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 80581 2022-11-23T04:06:02.5868405Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:06:02.5868843Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:06:02.5869420Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:06:02.5869873Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:06:02.5870299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:06:02.5870914Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:06:02.5871349Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:06:02.5871929Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:06:02.5872382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:06:02.5872806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:06:02.5873419Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:06:02.5873839Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:06:02.5874422Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:06:02.5874949Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:06:02.5875375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:06:02.5875993Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:06:02.5876425Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:06:02.5877005Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:06:02.5877444Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:06:02.5877873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:06:02.5878256Z skip: Need at least 4 CUDA devices (5.047s) 2022-11-23T04:06:02.5878437Z 2022-11-23T04:06:02.5878776Z ---------------------------------------------------------------------- 2022-11-23T04:06:02.5879102Z Ran 4 tests in 18.361s 2022-11-23T04:06:02.5879254Z 2022-11-23T04:06:02.5879355Z OK (skipped=4) 2022-11-23T04:06:02.5879500Z 2022-11-23T04:06:02.5879616Z Generating XML reports... 2022-11-23T04:06:02.5880259Z Generated XML report: test-reports/python-unittest/distributed._tensor.parallel.test_parallelize_api/TEST-TensorParallelAPITests-20221123040542.xml 2022-11-23T04:06:02.5880625Z 2022-11-23T04:06:02.5881095Z ##[endgroup] 2022-11-23T04:06:02.5881767Z FINISHED PRINTING LOG FILE of distributed/_tensor/parallel/test_parallelize_api (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-parallel-test_parallelize_api_31nqe95i) 2022-11-23T04:06:02.5882144Z 2022-11-23T04:06:02.5882403Z Running distributed/_shard/test_sharder ... [2022-11-23 04:06:02.575529] 2022-11-23T04:06:02.5883095Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/test_sharder.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 04:06:02.576197] 2022-11-23T04:06:06.4388718Z 2022-11-23T04:06:06.4390197Z Expand the folded group to see the log file of distributed/_shard/test_sharder 2022-11-23T04:06:06.4392643Z ##[group]PRINTING LOG FILE of distributed/_shard/test_sharder (/var/lib/jenkins/pytorch/test/test-reports/distributed-_shard-test_sharder_jb7hk318) 2022-11-23T04:06:06.4394401Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnt_qjlgz 2022-11-23T04:06:06.4395973Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnt_qjlgz/_remote_module_non_scriptable.py 2022-11-23T04:06:06.4396817Z 2022-11-23T04:06:06.4397651Z ##[endgroup] 2022-11-23T04:06:06.4399910Z FINISHED PRINTING LOG FILE of distributed/_shard/test_sharder (/var/lib/jenkins/pytorch/test/test-reports/distributed-_shard-test_sharder_jb7hk318) 2022-11-23T04:06:06.4400885Z 2022-11-23T04:06:06.4401938Z Running distributed/_shard/sharded_tensor/ops/test_tensor_ops ... [2022-11-23 04:06:06.439294] 2022-11-23T04:06:06.4404319Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_tensor_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 04:06:06.440014] 2022-11-23T04:06:32.7366728Z 2022-11-23T04:06:32.7370863Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_tensor_ops 2022-11-23T04:06:32.7373230Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_tensor_ops (/var/lib/jenkins/pytorch/test/test-reports/distributed-_shard-sharded_tensor-ops-test_tensor_ops__yvb0e94) 2022-11-23T04:06:32.7375879Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops 2022-11-23T04:06:32.7376876Z 2022-11-23T04:06:32.7377195Z Running tests... 2022-11-23T04:06:32.7379563Z ---------------------------------------------------------------------- 2022-11-23T04:06:32.7384353Z test_clone (__main__.TestTensorOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80979 2022-11-23T04:06:32.7386545Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80980 2022-11-23T04:06:32.7387853Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 80981 2022-11-23T04:06:32.7389131Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 80982 2022-11-23T04:06:32.7391435Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:06:32.7392762Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:06:32.7395322Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:06:32.7397177Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:06:32.7398724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:06:32.7401054Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:06:32.7402366Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:06:32.7404083Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:06:32.7405427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:06:32.7406670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:06:32.7408891Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:06:32.7410166Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:06:32.7412029Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:06:32.7413382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:06:32.7414636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:06:32.7416443Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:06:32.7417702Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:06:32.7419404Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:06:32.7420735Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:06:32.7421970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:06:32.7423088Z skip: Need at least 4 CUDA devices (4.741s) 2022-11-23T04:06:32.7424354Z test_deep_copy (__main__.TestTensorOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81247 2022-11-23T04:06:32.7425778Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81248 2022-11-23T04:06:32.7427044Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 81249 2022-11-23T04:06:32.7428317Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 81250 2022-11-23T04:06:32.7430108Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:06:32.7431384Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:06:32.7433084Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:06:32.7434399Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:06:32.7435620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:06:32.7437430Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:06:32.7438929Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:06:32.7440629Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:06:32.7441955Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:06:32.7443196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:06:32.7444998Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:06:32.7446242Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:06:32.7448235Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:06:32.7449576Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:06:32.7450986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:06:32.7452853Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:06:32.7454125Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:06:32.7455814Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:06:32.7457091Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:06:32.7457608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:06:32.7458022Z skip: Need at least 4 CUDA devices (5.040s) 2022-11-23T04:06:32.7458456Z test_detach (__main__.TestTensorOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81515 2022-11-23T04:06:32.7458945Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81516 2022-11-23T04:06:32.7459386Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 81517 2022-11-23T04:06:32.7459821Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 81518 2022-11-23T04:06:32.7460431Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:06:32.7460853Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:06:32.7461425Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:06:32.7461877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:06:32.7462298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:06:32.7462911Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:06:32.7463342Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:06:32.7463922Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:06:32.7464361Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:06:32.7464784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:06:32.7465398Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:06:32.7465827Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:06:32.7466404Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:06:32.7466858Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:06:32.7467281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:06:32.7467958Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:06:32.7468395Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:06:32.7468964Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:06:32.7469412Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:06:32.7469836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:06:32.7470216Z skip: Need at least 4 CUDA devices (4.139s) 2022-11-23T04:06:32.7470651Z test_inplace_copy (__main__.TestTensorOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81783 2022-11-23T04:06:32.7471129Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81784 2022-11-23T04:06:32.7471566Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 81785 2022-11-23T04:06:32.7472052Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 81786 2022-11-23T04:06:32.7472672Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:06:32.7473105Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:06:32.7473676Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:06:32.7474126Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:06:32.7474549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:06:32.7475147Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:06:32.7475580Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:06:32.7476156Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:06:32.7476611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:06:32.7477030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:06:32.7477641Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:06:32.7478072Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:06:32.7478632Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:06:32.7479088Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:06:32.7479510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:06:32.7480122Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:06:32.7480561Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:06:32.7481135Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:06:32.7481586Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:06:32.7481994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:06:32.7482379Z skip: Need at least 4 CUDA devices (4.039s) 2022-11-23T04:06:32.7482820Z test_set_requires_grad (__main__.TestTensorOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82051 2022-11-23T04:06:32.7483313Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82052 2022-11-23T04:06:32.7483749Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 82053 2022-11-23T04:06:32.7484185Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 82054 2022-11-23T04:06:32.7484860Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:06:32.7485281Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:06:32.7485855Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:06:32.7486304Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:06:32.7486726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:06:32.7487335Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:06:32.7487812Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:06:32.7488391Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:06:32.7489021Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:06:32.7489515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T04:06:32.7490260Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:06:32.7490780Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:06:32.7491469Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:06:32.7492006Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:06:32.7492512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:06:32.7493247Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:06:32.7493763Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:06:32.7494456Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:06:32.7495119Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:06:32.7495861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T04:06:32.7496470Z skip: Need at least 4 CUDA devices (4.439s) 2022-11-23T04:06:32.7496810Z 2022-11-23T04:06:32.7497253Z ---------------------------------------------------------------------- 2022-11-23T04:06:32.7497688Z Ran 5 tests in 22.399s 2022-11-23T04:06:32.7497888Z 2022-11-23T04:06:32.7498000Z OK (skipped=5) 2022-11-23T04:06:32.7498186Z 2022-11-23T04:06:32.7498331Z Generating XML reports... 2022-11-23T04:06:32.7499202Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops/TEST-TestTensorOps-20221123040608.xml 2022-11-23T04:06:32.7499711Z 2022-11-23T04:06:32.7500402Z ##[endgroup] 2022-11-23T04:06:32.7504464Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_tensor_ops (/var/lib/jenkins/pytorch/test/test-reports/distributed-_shard-sharded_tensor-ops-test_tensor_ops__yvb0e94) 2022-11-23T04:06:32.7505566Z 2022-11-23T04:06:32.7506353Z Running distributed/_composable/test_fully_shard ... [2022-11-23 04:06:32.736687] 2022-11-23T04:06:32.7508264Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_composable/test_fully_shard.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 04:06:32.737347] 2022-11-23T04:07:02.7830973Z 2022-11-23T04:07:02.7832313Z Expand the folded group to see the log file of distributed/_composable/test_fully_shard 2022-11-23T04:07:02.7834648Z ##[group]PRINTING LOG FILE of distributed/_composable/test_fully_shard (/var/lib/jenkins/pytorch/test/test-reports/distributed-_composable-test_fully_shard_pq57hr3o) 2022-11-23T04:07:02.7837093Z Test results will be stored in test-reports/python-unittest/distributed._composable.test_fully_shard 2022-11-23T04:07:02.7838513Z 2022-11-23T04:07:02.7838797Z Running tests... 2022-11-23T04:07:02.7839942Z ---------------------------------------------------------------------- 2022-11-23T04:07:02.7840994Z test_auto_wrap_policy (__main__.TestFSDPInitialization) 2022-11-23T04:07:02.7842377Z Tests passing an ``auto_wrap_policy``. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82386 2022-11-23T04:07:02.7843669Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82387 2022-11-23T04:07:02.7845365Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:07:02.7846506Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:07:02.7848230Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:07:02.7849714Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:07:02.7850904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:07:02.7852627Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:07:02.7853780Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:07:02.7855347Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:07:02.7856715Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:07:02.7857861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:07:02.7859635Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:07:02.7861473Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:07:02.7862992Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:07:02.7864309Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:07:02.7865454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:07:02.7866672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:07:02.7867579Z dist init r=1, world=2 2022-11-23T04:07:02.7868201Z dist init r=0, world=2 2022-11-23T04:07:02.7868818Z ok (5.832s) 2022-11-23T04:07:02.7869572Z test_device_id (__main__.TestFSDPInitialization) 2022-11-23T04:07:02.7870729Z Tests passing a ``device_id``. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82533 2022-11-23T04:07:02.7871982Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82534 2022-11-23T04:07:02.7873631Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:07:02.7874774Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:07:02.7876290Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:07:02.7877482Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:07:02.7878625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:07:02.7880285Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:07:02.7881432Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:07:02.7882960Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:07:02.7884153Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:07:02.7885491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:07:02.7887233Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:07:02.7889151Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:07:02.7890658Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:07:02.7891987Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:07:02.7893153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:07:02.7894373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:07:02.7895288Z dist init r=0, world=2 2022-11-23T04:07:02.7895930Z dist init r=1, world=2 2022-11-23T04:07:02.7896697Z ok (4.427s) 2022-11-23T04:07:02.7897524Z test_materialize_meta_module (__main__.TestFSDPInitialization) 2022-11-23T04:07:02.7899182Z Tests materializing a meta-device module. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82676 2022-11-23T04:07:02.7900496Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82677 2022-11-23T04:07:02.7902114Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:07:02.7903245Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:07:02.7904785Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:07:02.7905984Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:07:02.7907121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:07:02.7908798Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:07:02.7909945Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:07:02.7911477Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:07:02.7912651Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:07:02.7913800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:07:02.7915527Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:07:02.7917334Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:07:02.7918825Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:07:02.7920153Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:07:02.7921299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:07:02.7922513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:07:02.7923408Z dist init r=1, world=2 2022-11-23T04:07:02.7924048Z dist init r=0, world=2 2022-11-23T04:07:02.7924661Z ok (4.534s) 2022-11-23T04:07:02.7925453Z test_sync_module_states (__main__.TestFSDPInitialization) 2022-11-23T04:07:02.7926666Z Tests passing ``sync_module_states=True``. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82823 2022-11-23T04:07:02.7928276Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82824 2022-11-23T04:07:02.7929898Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:07:02.7931282Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:07:02.7932850Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:07:02.7934051Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:07:02.7934697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:07:02.7935459Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:07:02.7935986Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:07:02.7936670Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:07:02.7937215Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:07:02.7937809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:07:02.7938606Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:07:02.7939419Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:07:02.7940093Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:07:02.7940689Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:07:02.7941210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:07:02.7941746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:07:02.7942158Z dist init r=0, world=2 2022-11-23T04:07:02.7942456Z dist init r=1, world=2 2022-11-23T04:07:02.7942739Z ok (4.527s) 2022-11-23T04:07:02.7943059Z test_training (__main__.TestFSDPRuntime) 2022-11-23T04:07:02.7943607Z Tests training (forward, backward, optimizer). ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82970 2022-11-23T04:07:02.7944271Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82971 2022-11-23T04:07:02.7944956Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:07:02.7945394Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:07:02.7945973Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:07:02.7946428Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:07:02.7946866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T04:07:02.7947490Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T04:07:02.7947934Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T04:07:02.7948498Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T04:07:02.7948955Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T04:07:02.7949390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T04:07:02.7950041Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:07:02.7950719Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T04:07:02.7951285Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:07:02.7951784Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T04:07:02.7952226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T04:07:02.7952743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T04:07:02.7953093Z dist init r=1, world=2 2022-11-23T04:07:02.7953342Z dist init r=0, world=2 2022-11-23T04:07:02.7953583Z ok (6.737s) 2022-11-23T04:07:02.7953723Z 2022-11-23T04:07:02.7954004Z ---------------------------------------------------------------------- 2022-11-23T04:07:02.7954325Z Ran 5 tests in 26.059s 2022-11-23T04:07:02.7954465Z 2022-11-23T04:07:02.7954552Z OK 2022-11-23T04:07:02.7954678Z 2022-11-23T04:07:02.7954791Z Generating XML reports... 2022-11-23T04:07:02.7955436Z Generated XML report: test-reports/python-unittest/distributed._composable.test_fully_shard/TEST-TestFSDPInitialization-20221123040634.xml 2022-11-23T04:07:02.7956225Z Generated XML report: test-reports/python-unittest/distributed._composable.test_fully_shard/TEST-TestFSDPRuntime-20221123040634.xml 2022-11-23T04:07:02.7956547Z 2022-11-23T04:07:02.7957041Z ##[endgroup] 2022-11-23T04:07:02.7957685Z FINISHED PRINTING LOG FILE of distributed/_composable/test_fully_shard (/var/lib/jenkins/pytorch/test/test-reports/distributed-_composable-test_fully_shard_pq57hr3o) 2022-11-23T04:07:02.7958037Z 2022-11-23T04:07:02.7958323Z Running distributed/_composable/test_checkpoint ... [2022-11-23 04:07:02.783407] 2022-11-23T04:07:02.7959030Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_composable/test_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 04:07:02.784066] 2022-11-23T04:07:09.4369926Z 2022-11-23T04:07:09.4370896Z Expand the folded group to see the log file of distributed/_composable/test_checkpoint 2022-11-23T04:07:09.4373406Z ##[group]PRINTING LOG FILE of distributed/_composable/test_checkpoint (/var/lib/jenkins/pytorch/test/test-reports/distributed-_composable-test_checkpoint_kisjxrhn) 2022-11-23T04:07:09.4375877Z Test results will be stored in test-reports/python-unittest/distributed._composable.test_checkpoint 2022-11-23T04:07:09.4376733Z 2022-11-23T04:07:09.4377006Z Running tests... 2022-11-23T04:07:09.4378227Z ---------------------------------------------------------------------- 2022-11-23T04:07:09.4379363Z test_tensor_only_cpu (__main__.TestCheckpoint) ... ok (0.039s) 2022-11-23T04:07:09.4380467Z test_tensor_only_gpu (__main__.TestCheckpoint) ... ok (1.637s) 2022-11-23T04:07:09.4381072Z 2022-11-23T04:07:09.4381870Z ---------------------------------------------------------------------- 2022-11-23T04:07:09.4382802Z Ran 2 tests in 1.676s 2022-11-23T04:07:09.4383248Z 2022-11-23T04:07:09.4383488Z OK 2022-11-23T04:07:09.4383849Z 2022-11-23T04:07:09.4384161Z Generating XML reports... 2022-11-23T04:07:09.4385950Z Generated XML report: test-reports/python-unittest/distributed._composable.test_checkpoint/TEST-TestCheckpoint-20221123040705.xml 2022-11-23T04:07:09.4386914Z 2022-11-23T04:07:09.4387718Z ##[endgroup] 2022-11-23T04:07:09.4389580Z FINISHED PRINTING LOG FILE of distributed/_composable/test_checkpoint (/var/lib/jenkins/pytorch/test/test-reports/distributed-_composable-test_checkpoint_kisjxrhn) 2022-11-23T04:07:09.4390627Z 2022-11-23T04:07:11.7998751Z 2022-11-23T04:07:11.8000471Z real 150m26.564s 2022-11-23T04:07:11.8001213Z user 272m48.977s 2022-11-23T04:07:11.8001830Z sys 148m59.399s 2022-11-23T04:07:11.8002492Z + assert_git_not_dirty 2022-11-23T04:07:11.8003920Z + [[ linux-focal-rocm5.2-py3.8 != *rocm* ]] 2022-11-23T04:07:11.8004975Z + [[ linux-focal-rocm5.2-py3.8 == *cuda* ]] 2022-11-23T04:07:11.8005694Z + [[ 1 == 1 ]] 2022-11-23T04:07:11.8006267Z + test_rpc 2022-11-23T04:07:11.8007179Z + [[ linux-focal-rocm5.2-py3.8 != *rocm* ]] 2022-11-23T04:07:11.8151717Z ##[group]Run # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct 2022-11-23T04:07:11.8152747Z # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct 2022-11-23T04:07:11.8153815Z docker exec -t "7ae77914f0c0c5de7f89cc247b24f443680151b55b56bc99ab51a9510965bce3" sh -c "cd ../pytorch && sudo cp -R test/test-reports ../workspace/test" 2022-11-23T04:07:11.8192872Z shell: /bin/bash -e {0} 2022-11-23T04:07:11.8193324Z env: 2022-11-23T04:07:11.8193765Z GIT_DEFAULT_BRANCH: master 2022-11-23T04:07:11.8194338Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T04:07:11.8195219Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T04:07:11.8196180Z CONTAINER_NAME: 7ae77914f0c0c5de7f89cc247b24f443680151b55b56bc99ab51a9510965bce3 2022-11-23T04:07:11.8196772Z ##[endgroup] 2022-11-23T04:07:11.9435883Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2022-11-23T04:07:11.9897430Z Prepare all required actions 2022-11-23T04:07:11.9898724Z Getting action download info 2022-11-23T04:07:12.2337442Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2022-11-23T04:07:13.1495252Z ##[group]Run ./.github/actions/get-workflow-job-id 2022-11-23T04:07:13.1495603Z with: 2022-11-23T04:07:13.1496231Z github-token: *** 2022-11-23T04:07:13.1496498Z env: 2022-11-23T04:07:13.1496780Z GIT_DEFAULT_BRANCH: master 2022-11-23T04:07:13.1497136Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T04:07:13.1497695Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T04:07:13.1498305Z CONTAINER_NAME: 7ae77914f0c0c5de7f89cc247b24f443680151b55b56bc99ab51a9510965bce3 2022-11-23T04:07:13.1498692Z ##[endgroup] 2022-11-23T04:07:13.1532174Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2022-11-23T04:07:13.1532522Z with: 2022-11-23T04:07:13.1532776Z shell: bash 2022-11-23T04:07:13.1533053Z timeout_minutes: 10 2022-11-23T04:07:13.1533339Z max_attempts: 5 2022-11-23T04:07:13.1533624Z retry_wait_seconds: 30 2022-11-23T04:07:13.1534224Z command: set -eux python3 -m pip install requests==2.26.0 GHA_WORKFLOW_JOB_ID=$(python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}") echo "job-id=${GHA_WORKFLOW_JOB_ID}" >> "${GITHUB_OUTPUT}" 2022-11-23T04:07:13.1534820Z polling_interval_seconds: 1 2022-11-23T04:07:13.1535118Z warning_on_retry: true 2022-11-23T04:07:13.1535419Z continue_on_error: false 2022-11-23T04:07:13.1535695Z env: 2022-11-23T04:07:13.1535968Z GIT_DEFAULT_BRANCH: master 2022-11-23T04:07:13.1536316Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T04:07:13.1536858Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T04:07:13.1537443Z CONTAINER_NAME: 7ae77914f0c0c5de7f89cc247b24f443680151b55b56bc99ab51a9510965bce3 2022-11-23T04:07:13.1538083Z GITHUB_TOKEN: *** 2022-11-23T04:07:13.1538367Z ##[endgroup] 2022-11-23T04:07:13.2222146Z + python3 -m pip install requests==2.26.0 2022-11-23T04:07:14.1235706Z Collecting requests==2.26.0 2022-11-23T04:07:14.2486947Z Using cached https://files.pythonhosted.org/packages/92/96/144f70b972a9c0eabbd4391ef93ccd49d0f2747f4f6a2a2738e99e5adc65/requests-2.26.0-py2.py3-none-any.whl 2022-11-23T04:07:14.2659888Z Collecting idna<4,>=2.5; python_version >= "3" (from requests==2.26.0) 2022-11-23T04:07:14.2988138Z Using cached https://files.pythonhosted.org/packages/fc/34/3030de6f1370931b9dbb4dad48f6ab1015ab1d32447850b9fc94e60097be/idna-3.4-py3-none-any.whl 2022-11-23T04:07:14.3029225Z Collecting urllib3<1.27,>=1.21.1 (from requests==2.26.0) 2022-11-23T04:07:14.3725463Z Using cached https://files.pythonhosted.org/packages/6f/de/5be2e3eed8426f871b170663333a0f627fc2924cc386cd41be065e7ea870/urllib3-1.26.12-py2.py3-none-any.whl 2022-11-23T04:07:14.3965373Z Collecting certifi>=2017.4.17 (from requests==2.26.0) 2022-11-23T04:07:14.4426503Z Using cached https://files.pythonhosted.org/packages/1d/38/fa96a426e0c0e68aabc68e896584b83ad1eec779265a028e156ce509630e/certifi-2022.9.24-py3-none-any.whl 2022-11-23T04:07:14.4470008Z Collecting charset-normalizer~=2.0.0; python_version >= "3" (from requests==2.26.0) 2022-11-23T04:07:14.5756541Z Using cached https://files.pythonhosted.org/packages/06/b3/24afc8868eba069a7f03650ac750a778862dc34941a4bebeb58706715726/charset_normalizer-2.0.12-py3-none-any.whl 2022-11-23T04:07:14.5809418Z Installing collected packages: idna, urllib3, certifi, charset-normalizer, requests 2022-11-23T04:07:14.8040910Z Successfully installed certifi-2022.9.24 charset-normalizer-2.0.12 idna-3.4 requests-2.27.1 urllib3-1.26.12 2022-11-23T04:07:14.8471952Z ++ python3 .github/scripts/get_workflow_job_id.py 3528394938 worker-rocm-amd-94 2022-11-23T04:07:17.1021598Z + GHA_WORKFLOW_JOB_ID=9655437596 2022-11-23T04:07:17.1022789Z + echo job-id=9655437596 2022-11-23T04:07:17.2205396Z Command completed after 1 attempt(s). 2022-11-23T04:07:17.2428370Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2022-11-23T04:07:17.2428687Z kill "$MONITOR_SCRIPT_PID" 2022-11-23T04:07:17.2451983Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T04:07:17.2452254Z env: 2022-11-23T04:07:17.2452482Z GIT_DEFAULT_BRANCH: master 2022-11-23T04:07:17.2452773Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T04:07:17.2453232Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T04:07:17.2453725Z CONTAINER_NAME: 7ae77914f0c0c5de7f89cc247b24f443680151b55b56bc99ab51a9510965bce3 2022-11-23T04:07:17.2454058Z MONITOR_SCRIPT_PID: 6883 2022-11-23T04:07:17.2454292Z ##[endgroup] 2022-11-23T04:07:17.2616279Z Prepare all required actions 2022-11-23T04:07:17.2616618Z Getting action download info 2022-11-23T04:07:17.4867797Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2022-11-23T04:07:18.6249092Z Download action repository 'actions/upload-artifact@v3' (SHA:83fd05a356d7e2593de66fc9913b3002723633cb) 2022-11-23T04:07:19.5153468Z ##[group]Run ./.github/actions/upload-test-artifacts 2022-11-23T04:07:19.5153771Z with: 2022-11-23T04:07:19.5153990Z use-gha: true 2022-11-23T04:07:19.5154314Z file-suffix: test-distributed-1-2-linux.rocm.gpu_9655437596 2022-11-23T04:07:19.5154611Z env: 2022-11-23T04:07:19.5154832Z GIT_DEFAULT_BRANCH: master 2022-11-23T04:07:19.5155129Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T04:07:19.5155594Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T04:07:19.5156125Z CONTAINER_NAME: 7ae77914f0c0c5de7f89cc247b24f443680151b55b56bc99ab51a9510965bce3 2022-11-23T04:07:19.5156442Z ##[endgroup] 2022-11-23T04:07:19.5225634Z ##[group]Run actions/upload-artifact@v3 2022-11-23T04:07:19.5225896Z with: 2022-11-23T04:07:19.5226252Z name: test-jsons-runattempt1-test-distributed-1-2-linux.rocm.gpu_9655437596.zip 2022-11-23T04:07:19.5226628Z retention-days: 14 2022-11-23T04:07:19.5226883Z if-no-files-found: warn 2022-11-23T04:07:19.5227139Z path: test/**/*.json 2022-11-23T04:07:19.5227358Z env: 2022-11-23T04:07:19.5227584Z GIT_DEFAULT_BRANCH: master 2022-11-23T04:07:19.5227880Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T04:07:19.5228327Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T04:07:19.5228822Z CONTAINER_NAME: 7ae77914f0c0c5de7f89cc247b24f443680151b55b56bc99ab51a9510965bce3 2022-11-23T04:07:19.5229137Z ##[endgroup] 2022-11-23T04:07:19.8168597Z With the provided path, there will be 3 files uploaded 2022-11-23T04:07:19.8171582Z Starting artifact upload 2022-11-23T04:07:19.8173864Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging 2022-11-23T04:07:19.8175447Z Artifact name is valid! 2022-11-23T04:07:19.9371976Z Container for artifact "test-jsons-runattempt1-test-distributed-1-2-linux.rocm.gpu_9655437596.zip" successfully created. Starting upload of file(s) 2022-11-23T04:07:20.2957452Z Total size of all the files uploaded is 29304 bytes 2022-11-23T04:07:20.2958475Z File upload process has finished. Finalizing the artifact upload 2022-11-23T04:07:20.3820395Z Artifact has been finalized. All files have been successfully uploaded! 2022-11-23T04:07:20.3821019Z 2022-11-23T04:07:20.3821501Z The raw size of all the files that were specified for upload is 301646 bytes 2022-11-23T04:07:20.3822840Z The size of all the files that were uploaded is 29304 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage 2022-11-23T04:07:20.3823676Z 2022-11-23T04:07:20.3825155Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 2022-11-23T04:07:20.3826167Z 2022-11-23T04:07:20.3827301Z Artifact test-jsons-runattempt1-test-distributed-1-2-linux.rocm.gpu_9655437596.zip has been successfully uploaded! 2022-11-23T04:07:20.3963893Z ##[group]Run actions/upload-artifact@v3 2022-11-23T04:07:20.3964495Z with: 2022-11-23T04:07:20.3965327Z name: test-reports-runattempt1-test-distributed-1-2-linux.rocm.gpu_9655437596.zip 2022-11-23T04:07:20.3966213Z retention-days: 14 2022-11-23T04:07:20.3966779Z if-no-files-found: ignore 2022-11-23T04:07:20.3967394Z path: test/**/*.xml test/**/*.csv 2022-11-23T04:07:20.3968070Z env: 2022-11-23T04:07:20.3968619Z GIT_DEFAULT_BRANCH: master 2022-11-23T04:07:20.3969681Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T04:07:20.3970920Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T04:07:20.3972232Z CONTAINER_NAME: 7ae77914f0c0c5de7f89cc247b24f443680151b55b56bc99ab51a9510965bce3 2022-11-23T04:07:20.3973077Z ##[endgroup] 2022-11-23T04:07:20.7401377Z With the provided path, there will be 794 files uploaded 2022-11-23T04:07:20.7402082Z Starting artifact upload 2022-11-23T04:07:20.7403036Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging 2022-11-23T04:07:20.7403688Z Artifact name is valid! 2022-11-23T04:07:20.8809770Z Container for artifact "test-reports-runattempt1-test-distributed-1-2-linux.rocm.gpu_9655437596.zip" successfully created. Starting upload of file(s) 2022-11-23T04:07:30.8998717Z Total file count: 794 ---- Processed file #126 (15.8%) 2022-11-23T04:07:40.8993799Z Total file count: 794 ---- Processed file #258 (32.4%) 2022-11-23T04:07:50.9001981Z Total file count: 794 ---- Processed file #376 (47.3%) 2022-11-23T04:08:00.9010252Z Total file count: 794 ---- Processed file #495 (62.3%) 2022-11-23T04:08:10.9016212Z Total file count: 794 ---- Processed file #633 (79.7%) 2022-11-23T04:08:20.9015651Z Total file count: 794 ---- Processed file #769 (96.8%) 2022-11-23T04:08:22.7599279Z Total size of all the files uploaded is 245513 bytes 2022-11-23T04:08:22.7600314Z File upload process has finished. Finalizing the artifact upload 2022-11-23T04:08:22.8617992Z Artifact has been finalized. All files have been successfully uploaded! 2022-11-23T04:08:22.8618616Z 2022-11-23T04:08:22.8619094Z The raw size of all the files that were specified for upload is 543263 bytes 2022-11-23T04:08:22.8620429Z The size of all the files that were uploaded is 245513 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage 2022-11-23T04:08:22.8623852Z 2022-11-23T04:08:22.8625512Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 2022-11-23T04:08:22.8626524Z 2022-11-23T04:08:22.8627687Z Artifact test-reports-runattempt1-test-distributed-1-2-linux.rocm.gpu_9655437596.zip has been successfully uploaded! 2022-11-23T04:08:22.8817987Z ##[group]Run actions/upload-artifact@v3 2022-11-23T04:08:22.8818703Z with: 2022-11-23T04:08:22.8819658Z name: usage-log-runattempt1-test-distributed-1-2-linux.rocm.gpu_9655437596.zip 2022-11-23T04:08:22.8820669Z retention-days: 14 2022-11-23T04:08:22.8821335Z if-no-files-found: ignore 2022-11-23T04:08:22.8822024Z path: usage_log.txt 2022-11-23T04:08:22.8822628Z env: 2022-11-23T04:08:22.8823230Z GIT_DEFAULT_BRANCH: master 2022-11-23T04:08:22.8824011Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T04:08:22.8825238Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T04:08:22.8826562Z CONTAINER_NAME: 7ae77914f0c0c5de7f89cc247b24f443680151b55b56bc99ab51a9510965bce3 2022-11-23T04:08:22.8827384Z ##[endgroup] 2022-11-23T04:08:22.9472209Z With the provided path, there will be 1 file uploaded 2022-11-23T04:08:22.9473756Z Starting artifact upload 2022-11-23T04:08:22.9475879Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging 2022-11-23T04:08:22.9477344Z Artifact name is valid! 2022-11-23T04:08:23.0611455Z Container for artifact "usage-log-runattempt1-test-distributed-1-2-linux.rocm.gpu_9655437596.zip" successfully created. Starting upload of file(s) 2022-11-23T04:08:23.9578091Z Total size of all the files uploaded is 537457 bytes 2022-11-23T04:08:23.9579502Z File upload process has finished. Finalizing the artifact upload 2022-11-23T04:08:24.0706643Z Artifact has been finalized. All files have been successfully uploaded! 2022-11-23T04:08:24.0707591Z 2022-11-23T04:08:24.0708084Z The raw size of all the files that were specified for upload is 11188159 bytes 2022-11-23T04:08:24.0709401Z The size of all the files that were uploaded is 537457 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage 2022-11-23T04:08:24.0710239Z 2022-11-23T04:08:24.0712243Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 2022-11-23T04:08:24.0713682Z 2022-11-23T04:08:24.0715197Z Artifact usage-log-runattempt1-test-distributed-1-2-linux.rocm.gpu_9655437596.zip has been successfully uploaded! 2022-11-23T04:08:24.0833280Z ##[group]Run set -x 2022-11-23T04:08:24.0834006Z set -x 2022-11-23T04:08:24.0834783Z python3 -m pip install -r requirements.txt 2022-11-23T04:08:24.0835700Z python3 -m pip install boto3==1.19.12 2022-11-23T04:08:24.0836751Z python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test 2022-11-23T04:08:24.0891895Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T04:08:24.0892658Z env: 2022-11-23T04:08:24.0893299Z GIT_DEFAULT_BRANCH: master 2022-11-23T04:08:24.0894084Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T04:08:24.0895313Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T04:08:24.0896630Z CONTAINER_NAME: 7ae77914f0c0c5de7f89cc247b24f443680151b55b56bc99ab51a9510965bce3 2022-11-23T04:08:24.0897514Z AWS_DEFAULT_REGION: us-east-1 2022-11-23T04:08:24.0898181Z BRANCH: master 2022-11-23T04:08:24.0898879Z TEST_CONFIG: distributed 2022-11-23T04:08:24.0899521Z SHARD_NUMBER: 1 2022-11-23T04:08:24.0900280Z BUILD_ENVIRONMENT: linux-focal-rocm5.2-py3.8 2022-11-23T04:08:24.0901012Z PR_NUMBER: 2022-11-23T04:08:24.0901651Z PYTORCH_RETRY_TEST_CASES: 1 2022-11-23T04:08:24.0902373Z PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1 2022-11-23T04:08:24.0903013Z SHA1: 1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T04:08:24.0903485Z TAG: 2022-11-23T04:08:24.0903888Z WORKFLOW_ID: 3528394938 2022-11-23T04:08:24.0905108Z GITHUB_TOKEN: *** 2022-11-23T04:08:24.0905657Z AWS_ACCESS_KEY_ID: *** 2022-11-23T04:08:24.0906318Z AWS_SECRET_ACCESS_KEY: *** 2022-11-23T04:08:24.0906818Z GHA_WORKFLOW_JOB_ID: 9655437596 2022-11-23T04:08:24.0907267Z ##[endgroup] 2022-11-23T04:08:24.0975928Z + python3 -m pip install -r requirements.txt 2022-11-23T04:08:25.0261104Z Collecting astunparse (from -r requirements.txt (line 2)) 2022-11-23T04:08:25.0955276Z Using cached https://files.pythonhosted.org/packages/2b/03/13dde6512ad7b4557eb792fbcf0c653af6076b81e5941d36ec61f7ce6028/astunparse-1.6.3-py2.py3-none-any.whl 2022-11-23T04:08:25.1009095Z Collecting expecttest (from -r requirements.txt (line 3)) 2022-11-23T04:08:25.1507381Z Using cached https://files.pythonhosted.org/packages/a6/26/1a287e44618c14659db0256bc1ee239c2134f9c863cb9a85813ecab73413/expecttest-0.1.4-py3-none-any.whl 2022-11-23T04:08:25.1526115Z Collecting future (from -r requirements.txt (line 4)) 2022-11-23T04:08:25.2126152Z Collecting hypothesis (from -r requirements.txt (line 5)) 2022-11-23T04:08:26.0763090Z Using cached https://files.pythonhosted.org/packages/82/7e/01691560a3a98bb1ae909affde49392f599eef3a0b91c27b992bbbde2abb/hypothesis-6.31.6-py3-none-any.whl 2022-11-23T04:08:26.1366465Z Collecting numpy (from -r requirements.txt (line 6)) 2022-11-23T04:08:26.9506407Z Using cached https://files.pythonhosted.org/packages/45/b2/6c7545bb7a38754d63048c7696804a0d947328125d81bf12beaa692c3ae3/numpy-1.19.5-cp36-cp36m-manylinux1_x86_64.whl 2022-11-23T04:08:27.2600509Z Collecting psutil (from -r requirements.txt (line 7)) 2022-11-23T04:08:27.5494884Z Collecting pyyaml (from -r requirements.txt (line 8)) 2022-11-23T04:08:27.6644975Z Using cached https://files.pythonhosted.org/packages/b3/85/79b9e5b4e8d3c0ac657f4e8617713cca8408f6cdc65d2ee6554217cedff1/PyYAML-6.0-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl 2022-11-23T04:08:27.6795786Z Collecting requests (from -r requirements.txt (line 9)) 2022-11-23T04:08:27.7472288Z Using cached https://files.pythonhosted.org/packages/2d/61/08076519c80041bc0ffa1a8af0cbd3bf3e2b62af10435d269a9d0f40564d/requests-2.27.1-py2.py3-none-any.whl 2022-11-23T04:08:27.7640935Z Collecting setuptools (from -r requirements.txt (line 10)) 2022-11-23T04:08:28.2171157Z Using cached https://files.pythonhosted.org/packages/b0/3a/88b210db68e56854d0bcf4b38e165e03be377e13907746f825790f3df5bf/setuptools-59.6.0-py3-none-any.whl 2022-11-23T04:08:28.2831311Z Collecting six (from -r requirements.txt (line 11)) 2022-11-23T04:08:28.3162274Z Using cached https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl 2022-11-23T04:08:28.3185769Z Collecting types-dataclasses (from -r requirements.txt (line 12)) 2022-11-23T04:08:28.3697726Z Using cached https://files.pythonhosted.org/packages/31/85/23ab2bbc280266af5bf22ded4e070946d1694d1721ced90666b649eaa795/types_dataclasses-0.6.6-py3-none-any.whl 2022-11-23T04:08:28.3711946Z Collecting typing_extensions (from -r requirements.txt (line 13)) 2022-11-23T04:08:28.4358011Z Using cached https://files.pythonhosted.org/packages/45/6b/44f7f8f1e110027cf88956b59f2fad776cca7e1704396d043f89effd3a0e/typing_extensions-4.1.1-py3-none-any.whl 2022-11-23T04:08:28.4378734Z Collecting sympy (from -r requirements.txt (line 14)) 2022-11-23T04:08:28.5508530Z Using cached https://files.pythonhosted.org/packages/78/43/33c5a5e7fbafbf51520f4e09cb0634a1ca1d4cd5469c57967e43183d7a42/sympy-1.9-py3-none-any.whl 2022-11-23T04:08:28.8258993Z Collecting filelock (from -r requirements.txt (line 15)) 2022-11-23T04:08:28.8681285Z Using cached https://files.pythonhosted.org/packages/84/ce/8916d10ef537f3f3b046843255f9799504aa41862bfa87844b9bdc5361cd/filelock-3.4.1-py3-none-any.whl 2022-11-23T04:08:28.8784663Z Collecting networkx (from -r requirements.txt (line 16)) 2022-11-23T04:08:28.9646786Z Using cached https://files.pythonhosted.org/packages/f3/b7/c7f488101c0bb5e4178f3cde416004280fd40262433496830de8a8c21613/networkx-2.5.1-py3-none-any.whl 2022-11-23T04:08:29.0682827Z Collecting jinja2 (from -r requirements.txt (line 17)) 2022-11-23T04:08:29.1136477Z Using cached https://files.pythonhosted.org/packages/20/9a/e5d9ec41927401e41aea8af6d16e78b5e612bca4699d417f646a9610a076/Jinja2-3.0.3-py3-none-any.whl 2022-11-23T04:08:29.1226042Z Collecting wheel<1.0,>=0.23.0 (from astunparse->-r requirements.txt (line 2)) 2022-11-23T04:08:29.1843622Z Using cached https://files.pythonhosted.org/packages/27/d6/003e593296a85fd6ed616ed962795b2f87709c3eee2bca4f6d0fe55c6d00/wheel-0.37.1-py2.py3-none-any.whl 2022-11-23T04:08:29.1900095Z Collecting sortedcontainers<3.0.0,>=2.1.0 (from hypothesis->-r requirements.txt (line 5)) 2022-11-23T04:08:29.2259062Z Using cached https://files.pythonhosted.org/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl 2022-11-23T04:08:29.2287119Z Collecting attrs>=19.2.0 (from hypothesis->-r requirements.txt (line 5)) 2022-11-23T04:08:29.2650706Z Using cached https://files.pythonhosted.org/packages/f2/bc/d817287d1aa01878af07c19505fafd1165cd6a119e9d0821ca1d1c20312d/attrs-22.1.0-py2.py3-none-any.whl 2022-11-23T04:08:29.2985648Z Collecting certifi>=2017.4.17 (from requests->-r requirements.txt (line 9)) 2022-11-23T04:08:29.3325366Z Using cached https://files.pythonhosted.org/packages/1d/38/fa96a426e0c0e68aabc68e896584b83ad1eec779265a028e156ce509630e/certifi-2022.9.24-py3-none-any.whl 2022-11-23T04:08:29.3367113Z Collecting idna<4,>=2.5; python_version >= "3" (from requests->-r requirements.txt (line 9)) 2022-11-23T04:08:29.3570010Z Using cached https://files.pythonhosted.org/packages/fc/34/3030de6f1370931b9dbb4dad48f6ab1015ab1d32447850b9fc94e60097be/idna-3.4-py3-none-any.whl 2022-11-23T04:08:29.3608149Z Collecting urllib3<1.27,>=1.21.1 (from requests->-r requirements.txt (line 9)) 2022-11-23T04:08:29.4222612Z Using cached https://files.pythonhosted.org/packages/6f/de/5be2e3eed8426f871b170663333a0f627fc2924cc386cd41be065e7ea870/urllib3-1.26.12-py2.py3-none-any.whl 2022-11-23T04:08:29.4463895Z Collecting charset-normalizer~=2.0.0; python_version >= "3" (from requests->-r requirements.txt (line 9)) 2022-11-23T04:08:29.5680109Z Using cached https://files.pythonhosted.org/packages/06/b3/24afc8868eba069a7f03650ac750a778862dc34941a4bebeb58706715726/charset_normalizer-2.0.12-py3-none-any.whl 2022-11-23T04:08:29.5729360Z Collecting mpmath>=0.19 (from sympy->-r requirements.txt (line 14)) 2022-11-23T04:08:29.6138562Z Using cached https://files.pythonhosted.org/packages/d4/cf/3965bddbb4f1a61c49aacae0e78fd1fe36b5dc36c797b31f30cf07dcbbb7/mpmath-1.2.1-py3-none-any.whl 2022-11-23T04:08:29.6391860Z Collecting decorator<5,>=4.3 (from networkx->-r requirements.txt (line 16)) 2022-11-23T04:08:29.6757355Z Using cached https://files.pythonhosted.org/packages/ed/1b/72a1821152d07cf1d8b6fce298aeb06a7eb90f4d6d41acec9861e7cc6df0/decorator-4.4.2-py2.py3-none-any.whl 2022-11-23T04:08:29.6778271Z Collecting MarkupSafe>=2.0 (from jinja2->-r requirements.txt (line 17)) 2022-11-23T04:08:29.8035240Z Using cached https://files.pythonhosted.org/packages/fc/d6/57f9a97e56447a1e340f8574836d3b636e2c14de304943836bd645fa9c7e/MarkupSafe-2.0.1-cp36-cp36m-manylinux1_x86_64.whl 2022-11-23T04:08:29.8071447Z Installing collected packages: wheel, six, astunparse, expecttest, future, sortedcontainers, attrs, hypothesis, numpy, psutil, pyyaml, certifi, idna, urllib3, charset-normalizer, requests, setuptools, types-dataclasses, typing-extensions, mpmath, sympy, filelock, decorator, networkx, MarkupSafe, jinja2 2022-11-23T04:08:39.6940687Z Successfully installed MarkupSafe-2.0.1 astunparse-1.6.3 attrs-22.1.0 certifi-2022.9.24 charset-normalizer-2.0.12 decorator-4.4.2 expecttest-0.1.4 filelock-3.4.1 future-0.18.2 hypothesis-6.31.6 idna-3.4 jinja2-3.0.3 mpmath-1.2.1 networkx-2.5.1 numpy-1.19.5 psutil-5.9.4 pyyaml-6.0 requests-2.27.1 setuptools-59.6.0 six-1.16.0 sortedcontainers-2.4.0 sympy-1.9 types-dataclasses-0.6.6 typing-extensions-4.1.1 urllib3-1.26.12 wheel-0.37.1 2022-11-23T04:08:39.8671244Z + python3 -m pip install boto3==1.19.12 2022-11-23T04:08:40.7836752Z Collecting boto3==1.19.12 2022-11-23T04:08:41.5609289Z Using cached https://files.pythonhosted.org/packages/5e/e1/156846b09fca21b9b164c54200011e3bd17f29187cbfc6903a8e0281a304/boto3-1.19.12-py3-none-any.whl 2022-11-23T04:08:41.5767353Z Collecting botocore<1.23.0,>=1.22.12 (from boto3==1.19.12) 2022-11-23T04:08:42.6353873Z Using cached https://files.pythonhosted.org/packages/6a/73/552b27e3a1b4f83630907c4958be78e9d4c906e73efd554ebd5e21cb1692/botocore-1.22.12-py3-none-any.whl 2022-11-23T04:08:42.9451825Z Collecting s3transfer<0.6.0,>=0.5.0 (from boto3==1.19.12) 2022-11-23T04:08:42.9942999Z Using cached https://files.pythonhosted.org/packages/7b/9c/f51775ebe7df5a7aa4e7c79ed671bde94e154bd968aca8d65bb24aba0c8c/s3transfer-0.5.2-py3-none-any.whl 2022-11-23T04:08:43.0010095Z Collecting jmespath<1.0.0,>=0.7.1 (from boto3==1.19.12) 2022-11-23T04:08:43.0313749Z Using cached https://files.pythonhosted.org/packages/07/cb/5f001272b6faeb23c1c9e0acc04d48eaaf5c862c17709d20e3469c6e0139/jmespath-0.10.0-py2.py3-none-any.whl 2022-11-23T04:08:43.0349025Z Collecting python-dateutil<3.0.0,>=2.1 (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) 2022-11-23T04:08:43.0853175Z Using cached https://files.pythonhosted.org/packages/36/7a/87837f39d0296e723bb9b62bbb257d0355c7f6128853c78955f57342a56d/python_dateutil-2.8.2-py2.py3-none-any.whl 2022-11-23T04:08:43.0931912Z Collecting urllib3<1.27,>=1.25.4 (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) 2022-11-23T04:08:43.1505610Z Using cached https://files.pythonhosted.org/packages/6f/de/5be2e3eed8426f871b170663333a0f627fc2924cc386cd41be065e7ea870/urllib3-1.26.12-py2.py3-none-any.whl 2022-11-23T04:08:43.1743965Z Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) 2022-11-23T04:08:43.1949436Z Using cached https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl 2022-11-23T04:08:43.1973089Z Installing collected packages: six, python-dateutil, jmespath, urllib3, botocore, s3transfer, boto3 2022-11-23T04:08:43.8050578Z Successfully installed boto3-1.19.12 botocore-1.22.12 jmespath-0.10.0 python-dateutil-2.8.2 s3transfer-0.5.2 six-1.16.0 urllib3-1.26.12 2022-11-23T04:08:43.9002917Z + python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test 2022-11-23T04:08:54.2991815Z [scribe] Scribe access token not provided, sending report via boto3... 2022-11-23T04:08:54.2997064Z 2022-11-23T04:08:54.2998872Z ----- Historic stats comparison result ------ 2022-11-23T04:08:54.2999422Z 2022-11-23T04:08:54.3003884Z job: linux-focal-rocm5.2-py3.8 2022-11-23T04:08:54.3004769Z commit: 1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T04:08:54.3005228Z 2022-11-23T04:08:54.3005727Z Commit graph (base is most recent master ancestor with at least one S3 report): 2022-11-23T04:08:54.3006327Z 2022-11-23T04:08:54.3006562Z : (master) 2022-11-23T04:08:54.3007111Z | 2022-11-23T04:08:54.3016172Z * 1cfd3858ac (HEAD) total time 4676.33s 2022-11-23T04:08:54.3023897Z * 26322544b8 (base) 2 reports, total time 10567.84s � 1421.60s 2022-11-23T04:08:54.3025440Z * 7f4b4d2827 2 reports, total time 8461.32s � 1168.97s 2022-11-23T04:08:54.3026285Z * b50699f247 2 reports, total time 9450.96s � 2404.87s 2022-11-23T04:08:54.3027085Z * 8bf8e4d71e 2 reports, total time 9492.36s � 2525.06s 2022-11-23T04:08:54.3027926Z * ce856cee7e 2 reports, total time 10405.71s � 1187.54s 2022-11-23T04:08:54.3028750Z * 391b593ca2 2 reports, total time 10409.50s � 1113.28s 2022-11-23T04:08:54.3029581Z * 5bba783d21 2 reports, total time 10442.70s � 1177.08s 2022-11-23T04:08:54.3030375Z * ea920a1115 2 reports, total time 8384.52s � 1109.09s 2022-11-23T04:08:54.3031183Z * 74e62a1fef 2 reports, total time 8516.95s � 1120.62s 2022-11-23T04:08:54.3031998Z * 00b7d8ef23 2 reports, total time 10408.58s � 1157.84s 2022-11-23T04:08:54.3033005Z | 2022-11-23T04:08:54.3033373Z : 2022-11-23T04:08:54.3033606Z 2022-11-23T04:08:54.3033896Z Removed (across 518 suites) 0 tests, totaling 0.00s 2022-11-23T04:08:54.3034542Z Modified (across 0 suites) 0 tests, totaling 0.00s 2022-11-23T04:08:54.3035161Z Added (across 69 suites) 962 tests, totaling +5643.75s 2022-11-23T04:08:54.3589475Z ##[group]Run # Only stop the docker container we started since there might be multiple runners on this host. 2022-11-23T04:08:54.3590626Z # Only stop the docker container we started since there might be multiple runners on this host. 2022-11-23T04:08:54.3591643Z docker stop "7ae77914f0c0c5de7f89cc247b24f443680151b55b56bc99ab51a9510965bce3" || true 2022-11-23T04:08:54.3592482Z # Prune all of the docker containers. 2022-11-23T04:08:54.3593282Z # Might fail if a prune is already in progress by another runner. 2022-11-23T04:08:54.3594072Z docker container prune -f || true 2022-11-23T04:08:54.3594864Z # Prune everything docker if there are more than 10 images (~200GB). 2022-11-23T04:08:54.3595697Z # This is easier than using a time filter, e.g., "until=24h". 2022-11-23T04:08:54.3596529Z # Might fail if a prune is already in progress by another runner. 2022-11-23T04:08:54.3597358Z image_count=$(docker images | wc -l) 2022-11-23T04:08:54.3598015Z if [[ ${image_count} -gt 10 ]]; then 2022-11-23T04:08:54.3598653Z  echo "Purging all docker caches" 2022-11-23T04:08:54.3599321Z  docker system prune -af || true 2022-11-23T04:08:54.3599894Z else 2022-11-23T04:08:54.3600571Z  echo "Will not purge docker, only ${image_count} images found" 2022-11-23T04:08:54.3601233Z fi 2022-11-23T04:08:54.3642562Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T04:08:54.3643107Z env: 2022-11-23T04:08:54.3643535Z GIT_DEFAULT_BRANCH: master 2022-11-23T04:08:54.3644120Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T04:08:54.3644996Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T04:08:54.3645943Z CONTAINER_NAME: 7ae77914f0c0c5de7f89cc247b24f443680151b55b56bc99ab51a9510965bce3 2022-11-23T04:08:54.3646556Z ##[endgroup] 2022-11-23T04:08:54.8093152Z 7ae77914f0c0c5de7f89cc247b24f443680151b55b56bc99ab51a9510965bce3 2022-11-23T04:09:05.1562873Z Deleted Containers: 2022-11-23T04:09:05.1563976Z 7ae77914f0c0c5de7f89cc247b24f443680151b55b56bc99ab51a9510965bce3 2022-11-23T04:09:05.1564525Z 2022-11-23T04:09:05.1564851Z Total reclaimed space: 8.468GB 2022-11-23T04:09:05.2113093Z Will not purge docker, only 4 images found 2022-11-23T04:09:05.2190077Z Post job cleanup. 2022-11-23T04:09:05.2239643Z Post job cleanup. 2022-11-23T04:09:05.3493902Z [command]/usr/bin/git version 2022-11-23T04:09:05.3545566Z git version 2.35.1 2022-11-23T04:09:05.3600104Z Temporarily overriding HOME='/home/pytorchci/actions-runner/_work/_temp/92154907-e9a6-4fe7-8221-dea4c00c8eb1' before making global git config changes 2022-11-23T04:09:05.3601579Z Adding repository directory to the temporary git global config as a safe directory 2022-11-23T04:09:05.3605470Z [command]/usr/bin/git config --global --add safe.directory /home/pytorchci/actions-runner/_work/pytorch/pytorch 2022-11-23T04:09:05.3658919Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2022-11-23T04:09:05.3699778Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || : 2022-11-23T04:09:05.4080649Z Entering 'android/libs/fbjni' 2022-11-23T04:09:05.4138634Z Entering 'third_party/FP16' 2022-11-23T04:09:05.4199501Z Entering 'third_party/FXdiv' 2022-11-23T04:09:05.4257373Z Entering 'third_party/NNPACK' 2022-11-23T04:09:05.4320287Z Entering 'third_party/QNNPACK' 2022-11-23T04:09:05.4387011Z Entering 'third_party/VulkanMemoryAllocator' 2022-11-23T04:09:05.4453031Z Entering 'third_party/XNNPACK' 2022-11-23T04:09:05.4531850Z Entering 'third_party/benchmark' 2022-11-23T04:09:05.4600671Z Entering 'third_party/cpuinfo' 2022-11-23T04:09:05.4664323Z Entering 'third_party/cub' 2022-11-23T04:09:05.4732459Z Entering 'third_party/cudnn_frontend' 2022-11-23T04:09:05.4811056Z Entering 'third_party/cutlass' 2022-11-23T04:09:05.4892577Z Entering 'third_party/eigen' 2022-11-23T04:09:05.4962015Z Entering 'third_party/fbgemm' 2022-11-23T04:09:05.5029034Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-11-23T04:09:05.5096606Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T04:09:05.5159419Z Entering 'third_party/fbgemm/third_party/googletest' 2022-11-23T04:09:05.5222192Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T04:09:05.5289827Z Entering 'third_party/flatbuffers' 2022-11-23T04:09:05.5361520Z Entering 'third_party/fmt' 2022-11-23T04:09:05.5432493Z Entering 'third_party/foxi' 2022-11-23T04:09:05.5494840Z Entering 'third_party/gemmlowp/gemmlowp' 2022-11-23T04:09:05.5563307Z Entering 'third_party/gloo' 2022-11-23T04:09:05.5627439Z Entering 'third_party/googletest' 2022-11-23T04:09:05.5692870Z Entering 'third_party/ideep' 2022-11-23T04:09:05.5759744Z Entering 'third_party/ideep/mkl-dnn' 2022-11-23T04:09:05.5832503Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T04:09:05.5907288Z Entering 'third_party/ios-cmake' 2022-11-23T04:09:05.5975275Z Entering 'third_party/ittapi' 2022-11-23T04:09:05.6042820Z Entering 'third_party/kineto' 2022-11-23T04:09:05.6108839Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T04:09:05.6177157Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T04:09:05.6247932Z Entering 'third_party/nccl/nccl' 2022-11-23T04:09:05.6316098Z Entering 'third_party/neon2sse' 2022-11-23T04:09:05.6387144Z Entering 'third_party/nlohmann' 2022-11-23T04:09:05.6456416Z Entering 'third_party/onnx' 2022-11-23T04:09:05.6548394Z Entering 'third_party/onnx/third_party/benchmark' 2022-11-23T04:09:05.6607597Z Entering 'third_party/onnx/third_party/pybind11' 2022-11-23T04:09:05.6680953Z Entering 'third_party/onnx-tensorrt' 2022-11-23T04:09:05.6741701Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T04:09:05.6805567Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T04:09:05.6869661Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T04:09:05.6928711Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T04:09:05.7004196Z Entering 'third_party/pocketfft' 2022-11-23T04:09:05.7072599Z Entering 'third_party/protobuf' 2022-11-23T04:09:05.7149124Z Entering 'third_party/protobuf/third_party/benchmark' 2022-11-23T04:09:05.7215265Z Entering 'third_party/protobuf/third_party/googletest' 2022-11-23T04:09:05.7279686Z Entering 'third_party/psimd' 2022-11-23T04:09:05.7340373Z Entering 'third_party/pthreadpool' 2022-11-23T04:09:05.7409009Z Entering 'third_party/pybind11' 2022-11-23T04:09:05.7464622Z Entering 'third_party/python-enum' 2022-11-23T04:09:05.7531472Z Entering 'third_party/python-peachpy' 2022-11-23T04:09:05.7596492Z Entering 'third_party/python-six' 2022-11-23T04:09:05.7664379Z Entering 'third_party/sleef' 2022-11-23T04:09:05.7732965Z Entering 'third_party/tbb' 2022-11-23T04:09:05.7806236Z Entering 'third_party/tensorpipe' 2022-11-23T04:09:05.7868756Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-11-23T04:09:05.7923023Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-11-23T04:09:05.7972641Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-11-23T04:09:05.8039478Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T04:09:05.8108225Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T04:09:05.8173827Z Entering 'third_party/zstd' 2022-11-23T04:09:05.8275409Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2022-11-23T04:09:05.8331840Z http.https://github.com/.extraheader 2022-11-23T04:09:05.8354648Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2022-11-23T04:09:05.8425292Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || : 2022-11-23T04:09:05.8838956Z Entering 'android/libs/fbjni' 2022-11-23T04:09:05.8868656Z http.https://github.com/.extraheader 2022-11-23T04:09:05.8929030Z Entering 'third_party/FP16' 2022-11-23T04:09:05.8965105Z http.https://github.com/.extraheader 2022-11-23T04:09:05.9029253Z Entering 'third_party/FXdiv' 2022-11-23T04:09:05.9063666Z http.https://github.com/.extraheader 2022-11-23T04:09:05.9112767Z Entering 'third_party/NNPACK' 2022-11-23T04:09:05.9148404Z http.https://github.com/.extraheader 2022-11-23T04:09:05.9195768Z Entering 'third_party/QNNPACK' 2022-11-23T04:09:05.9231598Z http.https://github.com/.extraheader 2022-11-23T04:09:05.9290235Z Entering 'third_party/VulkanMemoryAllocator' 2022-11-23T04:09:05.9320264Z http.https://github.com/.extraheader 2022-11-23T04:09:05.9380534Z Entering 'third_party/XNNPACK' 2022-11-23T04:09:05.9412049Z http.https://github.com/.extraheader 2022-11-23T04:09:05.9477254Z Entering 'third_party/benchmark' 2022-11-23T04:09:05.9510383Z http.https://github.com/.extraheader 2022-11-23T04:09:05.9571060Z Entering 'third_party/cpuinfo' 2022-11-23T04:09:05.9606047Z http.https://github.com/.extraheader 2022-11-23T04:09:05.9671014Z Entering 'third_party/cub' 2022-11-23T04:09:05.9702596Z http.https://github.com/.extraheader 2022-11-23T04:09:05.9763063Z Entering 'third_party/cudnn_frontend' 2022-11-23T04:09:05.9797577Z http.https://github.com/.extraheader 2022-11-23T04:09:05.9866584Z Entering 'third_party/cutlass' 2022-11-23T04:09:05.9897824Z http.https://github.com/.extraheader 2022-11-23T04:09:05.9970874Z Entering 'third_party/eigen' 2022-11-23T04:09:05.9996778Z http.https://github.com/.extraheader 2022-11-23T04:09:06.0053091Z Entering 'third_party/fbgemm' 2022-11-23T04:09:06.0084673Z http.https://github.com/.extraheader 2022-11-23T04:09:06.0136793Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-11-23T04:09:06.0172809Z http.https://github.com/.extraheader 2022-11-23T04:09:06.0233522Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T04:09:06.0265957Z http.https://github.com/.extraheader 2022-11-23T04:09:06.0326678Z Entering 'third_party/fbgemm/third_party/googletest' 2022-11-23T04:09:06.0362146Z http.https://github.com/.extraheader 2022-11-23T04:09:06.0414056Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T04:09:06.0449051Z http.https://github.com/.extraheader 2022-11-23T04:09:06.0513645Z Entering 'third_party/flatbuffers' 2022-11-23T04:09:06.0543661Z http.https://github.com/.extraheader 2022-11-23T04:09:06.0604448Z Entering 'third_party/fmt' 2022-11-23T04:09:06.0632958Z http.https://github.com/.extraheader 2022-11-23T04:09:06.0686139Z Entering 'third_party/foxi' 2022-11-23T04:09:06.0722271Z http.https://github.com/.extraheader 2022-11-23T04:09:06.0777809Z Entering 'third_party/gemmlowp/gemmlowp' 2022-11-23T04:09:06.0814368Z http.https://github.com/.extraheader 2022-11-23T04:09:06.0872141Z Entering 'third_party/gloo' 2022-11-23T04:09:06.0906303Z http.https://github.com/.extraheader 2022-11-23T04:09:06.0962936Z Entering 'third_party/googletest' 2022-11-23T04:09:06.0998600Z http.https://github.com/.extraheader 2022-11-23T04:09:06.1052703Z Entering 'third_party/ideep' 2022-11-23T04:09:06.1083075Z http.https://github.com/.extraheader 2022-11-23T04:09:06.1133792Z Entering 'third_party/ideep/mkl-dnn' 2022-11-23T04:09:06.1165513Z http.https://github.com/.extraheader 2022-11-23T04:09:06.1212294Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T04:09:06.1247322Z http.https://github.com/.extraheader 2022-11-23T04:09:06.1319934Z Entering 'third_party/ios-cmake' 2022-11-23T04:09:06.1352702Z http.https://github.com/.extraheader 2022-11-23T04:09:06.1410808Z Entering 'third_party/ittapi' 2022-11-23T04:09:06.1445910Z http.https://github.com/.extraheader 2022-11-23T04:09:06.1502382Z Entering 'third_party/kineto' 2022-11-23T04:09:06.1533372Z http.https://github.com/.extraheader 2022-11-23T04:09:06.1587023Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T04:09:06.1631542Z http.https://github.com/.extraheader 2022-11-23T04:09:06.1679488Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T04:09:06.1707370Z http.https://github.com/.extraheader 2022-11-23T04:09:06.1770576Z Entering 'third_party/nccl/nccl' 2022-11-23T04:09:06.1805339Z http.https://github.com/.extraheader 2022-11-23T04:09:06.1867673Z Entering 'third_party/neon2sse' 2022-11-23T04:09:06.1901847Z http.https://github.com/.extraheader 2022-11-23T04:09:06.1958328Z Entering 'third_party/nlohmann' 2022-11-23T04:09:06.1990765Z http.https://github.com/.extraheader 2022-11-23T04:09:06.2050682Z Entering 'third_party/onnx' 2022-11-23T04:09:06.2086745Z http.https://github.com/.extraheader 2022-11-23T04:09:06.2175523Z Entering 'third_party/onnx/third_party/benchmark' 2022-11-23T04:09:06.2210813Z http.https://github.com/.extraheader 2022-11-23T04:09:06.2274948Z Entering 'third_party/onnx/third_party/pybind11' 2022-11-23T04:09:06.2308128Z http.https://github.com/.extraheader 2022-11-23T04:09:06.2365865Z Entering 'third_party/onnx-tensorrt' 2022-11-23T04:09:06.2401723Z http.https://github.com/.extraheader 2022-11-23T04:09:06.2457569Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T04:09:06.2492957Z http.https://github.com/.extraheader 2022-11-23T04:09:06.2565989Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T04:09:06.2602157Z http.https://github.com/.extraheader 2022-11-23T04:09:06.2664345Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T04:09:06.2691716Z http.https://github.com/.extraheader 2022-11-23T04:09:06.2746270Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T04:09:06.2774832Z http.https://github.com/.extraheader 2022-11-23T04:09:06.2839151Z Entering 'third_party/pocketfft' 2022-11-23T04:09:06.2875613Z http.https://github.com/.extraheader 2022-11-23T04:09:06.2934527Z Entering 'third_party/protobuf' 2022-11-23T04:09:06.2970649Z http.https://github.com/.extraheader 2022-11-23T04:09:06.3038444Z Entering 'third_party/protobuf/third_party/benchmark' 2022-11-23T04:09:06.3075703Z http.https://github.com/.extraheader 2022-11-23T04:09:06.3131727Z Entering 'third_party/protobuf/third_party/googletest' 2022-11-23T04:09:06.3165825Z http.https://github.com/.extraheader 2022-11-23T04:09:06.3233859Z Entering 'third_party/psimd' 2022-11-23T04:09:06.3263935Z http.https://github.com/.extraheader 2022-11-23T04:09:06.3315321Z Entering 'third_party/pthreadpool' 2022-11-23T04:09:06.3351797Z http.https://github.com/.extraheader 2022-11-23T04:09:06.3396501Z Entering 'third_party/pybind11' 2022-11-23T04:09:06.3428055Z http.https://github.com/.extraheader 2022-11-23T04:09:06.3489413Z Entering 'third_party/python-enum' 2022-11-23T04:09:06.3523034Z http.https://github.com/.extraheader 2022-11-23T04:09:06.3581433Z Entering 'third_party/python-peachpy' 2022-11-23T04:09:06.3616786Z http.https://github.com/.extraheader 2022-11-23T04:09:06.3669907Z Entering 'third_party/python-six' 2022-11-23T04:09:06.3707207Z http.https://github.com/.extraheader 2022-11-23T04:09:06.3758600Z Entering 'third_party/sleef' 2022-11-23T04:09:06.3786892Z http.https://github.com/.extraheader 2022-11-23T04:09:06.3845276Z Entering 'third_party/tbb' 2022-11-23T04:09:06.3882804Z http.https://github.com/.extraheader 2022-11-23T04:09:06.3945352Z Entering 'third_party/tensorpipe' 2022-11-23T04:09:06.3980693Z http.https://github.com/.extraheader 2022-11-23T04:09:06.4040504Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-11-23T04:09:06.4070761Z http.https://github.com/.extraheader 2022-11-23T04:09:06.4130707Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-11-23T04:09:06.4159277Z http.https://github.com/.extraheader 2022-11-23T04:09:06.4213010Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-11-23T04:09:06.4248232Z http.https://github.com/.extraheader 2022-11-23T04:09:06.4304507Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T04:09:06.4338835Z http.https://github.com/.extraheader 2022-11-23T04:09:06.4394904Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T04:09:06.4426958Z http.https://github.com/.extraheader 2022-11-23T04:09:06.4488097Z Entering 'third_party/zstd' 2022-11-23T04:09:06.4523538Z http.https://github.com/.extraheader 2022-11-23T04:09:06.5027665Z Cleaning up orphan processes